Methods and system for distributing data to technical computing workers

Abstract

A method and system is disclosed for providing a distributed technical computing environment for distributing technical computing tasks from a technical computing client to technical computing workers for execution of the tasks on one or more computers systems. Tasks can be defined on a technical computing client, and the tasks organized into jobs. The technical computing client can directly distribute tasks to one or more technical computing workers. Furthermore, the technical computing client can submit tasks, or jobs comprising tasks, to an automatic task distribution mechanism that distributes the tasks automatically to one or more technical computing workers providing technical computing services. The technical computing worker performs technical computing of tasks and the results of the execution of tasks may be provided to the technical computing client. Data associated with the tasks is managed by a programmable interface associated with a data storage repository. The interface allows the various entities of the distributed technical computing environment to access data services performable by the interface or by a file system or a database and database management system associated with the data.

1 . In a distributed environment having a plurality of computational engines, a method of transferring data from a first of the computational engines to a second of the computational engines, the method comprising the steps of: 
 instructing an interface associated with a data repository to transfer to the second of the computational engines data associated with a task distributed from the first of the computational engines; and    transferring the data from the interface to the second of the computational engines,    whereby the interface in response to the instructions sends a request to the data repository to retrieve the data associated with the task and when the data repository returns the requested data the interface transfers the data in a format understandable by the second of the computational engines.    
     
     
         2 . The method of  claim 1 , further comprising the step of communicating with the interface to establish a communications channel with the data repository.  
     
     
         3 . The method of  claim 1 , wherein the step of instructing invokes a method of the interface to perform a pre-defined operation.  
     
     
         4 . The method of  claim 3 , wherein the method performs the step of taking an action to notify an entity of receipt of the data by the interface.  
     
     
         5 . The method of  claim 3 , wherein the method performs the step of taking an action to maintain integrity of the data.  
     
     
         6 . The method of  claim 3 , wherein the method performs the step of taking an action to update a data structure held by the data repository.  
     
     
         7 . The method of  claim 6 , wherein the step of taking an action includes the step of sending a query language command to a database management system managing the data structure.  
     
     
         8 . The method of  claim 6 , wherein the step of taking an action includes the step of calling a function associated with the interface.  
     
     
         9 . The method of  claim 1 , wherein the distributed environment comprises a distributed technical computing environment  
     
     
         10 . A system for distributing data in a computing environment, the system comprising: 
 a first computational engine submitting a task to distribute for processing in a computing environment, the task comprising at least one action to perform by the computing environment;    a data storage mechanism for holding data associated with the task in communication with the computing environment; and    An interface configured to manage transfer of the data associated with the task between the data storage mechanism and a second computational performing the processing in the distributed environment.    
     
     
         11 . The system of  claim 10 , further comprising an automatic task distribution mechanism in communication with the technical computing client, the automatic task distribution mechanism receiving the task submitted by the first computational engine and making the task available to one or more computational engines associated with the distributed technical computing environment.  
     
     
         12 . The system of  claim 10 , wherein the interface manages transfer of the data by notifying selected entities associated with the distributed environment of an event relating to the task.  
     
     
         13 . The system of  claim 12 , wherein the event comprises one of a change in a data value, a change in a data type, a change in a dimension of the data, receipt of the data by the interface, or transfer of the data from the interface to one of the computational engines.  
     
     
         14 . The system of  claim 13 , wherein the interface manages transfer of the data by providing data integrity.  
     
     
         15 . The system of  claim 1   1 , wherein the computing environment comprises a distributed technical computing environment.  
     
     
         16 . A computer readable medium holding computer executable instructions for performing a method of transferring data from a first computational engine to a second computational engine in a distributed environment having a plurality of computational engines, the method comprising the steps of: 
 instructing an interface associated with a data repository to transfer to the second of the computational devices data associated with a task distributed from the first of the computational devices; and    transferring the data from the interface to the second of the computational devices    whereby the interface in response to the instructions sends a request to the data repository to retrieve the data associated with the task and when the data repository returns the requested data the interface transfers the data in a format understandable by the second of the computational devices.    
     
     
         17 . The medium of  claim 15 , further comprising the step of communicating with the interface to establish a communications channel with the data repository.  
     
     
         18 . The medium of  claim 15 , wherein the step of instructing invokes a method of the interface to perform a pre-defined operation.  
     
     
         19 . The medium of  claim 15 , wherein the method performs the step of taking an action to notify an entity of receipt of the data by the interface.  
     
     
         20 . The medium of  claim 15 , wherein the method performs the step of taking an action to maintain integrity of the data.  
     
     
         21 . The medium of  claim 15 , wherein the method performs the step of taking an action to update a data structure held by the data repository.  
     
     
         22 . The medium of  claim 21 , wherein the step of taking an action includes the step of sending a query language command to a database management system managing the data structure.  
     
     
         23 . The medium of  claim 21 , wherein the step of taking an action includes the step of calling a function associated with the interface.  
     
     
         24 . The medium of  claim 11 , wherein the distributed environment comprises a distributed technical computing environment.  
     
     
         25 . In a distributed environment having a plurality of computational engines, a method of managing data transferred between a data repository associated with the distributed environment and one of the plurality of computational engines, the method comprising the steps of: 
 registering with a registry associated with the data repository to receive notification when a predefined action occurs with a task distributed from a first of the computational engines to a second of the computational engines; and    receiving notification when the predefined action occurs.    
     
     
         26 . The method of  claim 25 , wherein a scheduler of the distributed environment registers with the registry to receive the notification.  
     
     
         27 . The method of  claim 25 , wherein an interface associated with the data repository constructs the notification.  
     
     
         28 . The method of  claim 25  further comprising the step of, controlling the transfer of data between the data repository associated with the distributed environment and one of the plurality of computational engines based on content of the notification received.  
     
     
         29 . The method of  claim 25 , wherein the predefined action relates to data associated with the task.  
     
     
         30 . The method of  claim 26 , wherein the distributed environment comprises a distributed technical computing environment.  
     
     
         31 . A device readable medium holding device readable instructions for performing a method in a distributed environment having a plurality of computational engines, the method manages data transferred between a data repository associated with the distributed environment and one of the plurality of computational engines, the method comprising the steps of: 
 registering with a registry associated with the data repository to receive notification when a predefined action occurs with a task distributed from a first of the computational engines to a second of the computational engines; and    receiving notification when the predefined action occurs.    
     
     
         32 . The medium of  claim 31 , wherein a scheduler of the distributed technical computing environment registers with the registry to receive the notification.  
     
     
         33 . The medium of  claim 31 , wherein an interface associated with the data repository constructs the notification.  
     
     
         34 . The medium of  claim 31  further comprising the step of, controlling the transfer of data between the data repository associated with the distributed environment and one of the plurality of computational engines based on content of the notification received.  
     
     
         35 . The medium of  claim 31 , wherein the predefined action relates to data associated with the task.  
     
     
         36 . The medium of  claim 31 , wherein the distributed environment comprises a distributed technical computing environment.  
     
     
         37 . A system for managing data in a distributed environment, the system comprising: 
 a first computational engine submitting a task to distribute for processing in a computing environment, the task comprising at least one action to perform by the computing environment;    a data storage mechanism for holding data associated with the task in communication with the computing environment; and    an interface configured to notify an entity of the computing environment when an event occurs with the task,    wherein notification of the event from the interface provides the entity with an indication of a state of the task or a state of data associated with the task.    
     
     
         38 . The system of  claim 37 , further comprising an automatic task distribution mechanism in communication with the computational engine, the automatic task distribution mechanism receiving the task submitted by the computational engine , and making the task available to one or more other computational engines associated with the computing environment.  
     
     
         39 . The system of  claim 38 , wherein the event comprises one of a change in a data value, a change in a data type, a change in a dimension of the data, receipt of the data by the interface, transfer of the data from the interface to one of the computational engines, or a state of the processing of the task.  
     
     
         40 . The system of  claim 37 , wherein the distributed environment comprises a distributed technical computing environment.  
     
     
         41 . In a distributed environment having a plurality of computational engines, a method of managing distribution of tasks amongst the plurality of computational engines in the distributed environment, the method comprising the steps of: 
 receiving a task from a first of the computational engines, forwarding the task to a second of the computational engines,    receiving notification from an interface to a data repository for holding data associated with the task a predefined event occurred that concerns the task, and    taking an action to manage the distribution of tasks amongst the plurality of computational engines based on the content of the notification.    
     
     
         42 . The method of  claim 41 , wherein the step of taking an action comprises the step of forwarding the task to the second of the computational engines a second time.  
     
     
         43 . The method of  claim 41 , wherein the step of taking an action comprises the step of forwarding the task to a third of the computation engines.  
     
     
         44 . The method of  claim 41 , wherein the step of taking an action comprises the step of sending a result of the task to the first of the computational engines.  
     
     
         45 . The method of  claim 41 , wherein the step of taking an action comprises the step of forwarding other tasks to a third of the computational engines.  
     
     
         46 . The method of  claim 41 , wherein the step of taking an action comprises the step of forwarding other tasks to other computational engines from the plurality of the computational engines.  
     
     
         47 . The method of  claim 41 , wherein the distributed environment comprises a distributed technical computing environment.  
     
     
         48 . A device readable medium holding device executable instructions for performing a method in a distributed environment having a plurality of computational engines, the method for managing distribution of tasks amongst the plurality of computational engines in the distributed environment, the method comprising the steps of: 
 receiving a task from a first of the computational engines, forwarding the task to a second of the computational engines,    receiving notification from an interface to a data repository for holding data associated with the task a predefined event occurred that concerns the task, and    taking an action to manage the distribution of tasks amongst the plurality of computational devices based on the content of the notification.    
     
     
         49 . The medium of  claim 48 , wherein the step of taking an action comprises the step of forwarding the task to the second of the computational engines a second time.  
     
     
         50 . The medium of  claim 48 , wherein the step of taking an action comprises the step of forwarding the task to a third of the computation engines.  
     
     
         51 . The medium of  claim 48 , wherein the step of taking an action comprises the step of sending a result of the task to the first of the computational engines.  
     
     
         52 . The medium of  claim 48 , wherein the step of taking an action comprises the step of forwarding other tasks to a third of the computational engines.  
     
     
         53 . The medium of  claim 48 , wherein the step of taking an action comprises the step of forwarding other tasks to other computational devices from the plurality of the computational engines.  
     
     
         54 . The method of  claim 48 , wherein the distributed environment comprises a distributed technical computing environment.  
     
     
         55 . A system for managing distribution of tasks in a distributed environment, the system comprising: 
 a first computational engine submitting a task to distribute for processing in a computing environment, the task comprising at least one action to perform by the computing environment;    a data storage mechanism for holding data associated with the task in communication with the computing environment;    an automatic task distribution mechanism in communication with the first computational engine, the automatic task distribution mechanism receiving the task submitted by the technical computing client, and making the task available to one or more other computational engines associated with the computing environment; and    an interface configured to notify an automatic task distribution mechanism of the distributed environment when an event occurs with the task;    wherein the automatic task distribution mechanism in response to content contained in the notice form the interface takes an action to manage distribution of tasks in the computing environment.    
     
     
         56 . The system of  claim 55 , wherein the event comprises one of a change in a data value, a change in a data type, a change in a dimension of the data, receipt of the data by the interface, transfer of the data from the interface to one of the computational engines or a state of the processing of the task.  
     
     
         57 . The system of  claim 55 , wherein the interface takes an action to forward the task to the second of the computational engines a second time.  
     
     
         58 . The system of  claim 55 , wherein the interface takes an action to forward the task to a third of the computation engines.  
     
     
         59 . The system of  claim 55 , wherein the interface takes an action to send a result of the task to the first of the computational engines.  
     
     
         60 . The system of  claim 55 , wherein the interface takes an action to forward other tasks to a third of the computational engines.  
     
     
         61 . The system of  claim 55 , wherein the interface takes an action to forward other tasks to other computational engines from the plurality of the computational engines.  
     
     
         62 . The method of  claim 55 , wherein the distributed environment comprises a distributed technical computing environment.