Evolving parallel system to automatically improve the performance of multiple concurrent tasks on large datasets

Abstract

We describe a high-level computational framework especially well suited to parallel operations on large datasets. In a system in accordance with this framework, there is at least one, and generally several, instances of an architecture deployment as further described. We use the term “architecture deployment” herein to mean a cooperating group of processes together with the hardware on which the processes are executed. This is not to imply a one-to-one association of any process to particular hardware. To the contrary, as detailed below, an architecture deployment may dynamically spawn another deployment as appropriate, including provisioning needed hardware. The active architecture deployments together form a system that dynamically processes jobs requested by a user-customer, in accordance with customer's monetary budget and other criteria, in a robust and automatically scalable environment.

1 . A method comprising:
 deploying a first architecture deployment instance comprising a first plurality of software processes executing on a first set of hardware resources;   providing a supervisor process among the first plurality of processes to oversee at least the first plurality of processes;   providing a user interface process among the first plurality of processes to interface with a user to upload a collection of raw user data;   providing a data analysis process configured to analyze uploaded user data to infer its format and convert the user data into a dataset;   providing a model builder process arranged to construct a decision tree model based on a selected dataset; and   wherein each of said processes is arranged to communicate with at least some of the other processes to accomplish a job requested by a user via the user interface process.   
     
     
         2 . The method of  claim 1  wherein the first plurality of software processes communicate by means of a central software blackboard process implemented by the supervisor process. 
     
     
         3 . The method of  claim 2  and further comprising providing a prediction process among the first plurality of processes to provide a requested prediction based on a selected decision tree model. 
     
     
         4 . The method of  claim 3  and further wherein the first architecture deployment instance stores data representing respective customer budgets for a least one customer. 
     
     
         5 . The method of  claim 3  and further comprising identifying those tasks the first architecture deployment instance is performing that can be easily parallelizable and those tasks that can accelerate a current user job if additional computational resources were used. 
     
     
         6 . The method of  claim 5  and further comprising selecting one of four primitives, namely Auto-Replicate, Auto-Distribute, Auto-Duplicate, and Auto-Allocate for variously scaling selected aspects of the first architecture deployment instance. 
     
     
         7 . The method of  claim 5  and further comprising:
 determining whether the selected primitive would incur a cost in excess of the corresponding customer budget allocated to the current user job; and if not, executing the selected primitive operation. 
 
     
     
         8 . The method of  claim 5  and further wherein the first architecture deployment instance is configured to implement at least the following features:
 (a) at least one Action, 
 (b) at least one Reaction, 
 (c) at least one Goal, and 
 (d) a corresponding set of Requirements that express the hardware, operating systems, services, libraries and tools necessary to execute the said Actions and Reactions. 
 
     
     
         9 . The method of  claim 8  and wherein the deployment instance further includes:
 a set of constraints that define and constrain selected operating parameters; and 
 an embedded monitor that tracks the execution times and performance for each action and reaction for all the tasks that are being executed in the instance. 
 
     
     
         10 . The method of  claim 9  and further comprising:
 executing a selected one of the primitives, wherein said executing a primitive includes partitioning a set of pending actions into two disjoint sets; 
 spawning a second architecture deployment instance; 
 selectively distributing a subset of the pending actions to the second architecture deployment instance; 
 spawning a third architecture deployment instance; and 
 distributing the remaining pending actions to the third architecture deployment instance.