System and method of data partitioning for parallel processing of dynamically generated application data

Abstract

An improved system and method of data partitioning for parallel processing of dynamically generated application data is provided. An application may send a request to partition the application data specified by a data partitioning policy and to process each of the data partitions according to processing instructions. The data partitioning policy may be flexibly defined by an application for partitioning data any number of ways, including balancing the data volume across each of the partitions or partitioning the data by data type. Asynchronous data partition processors may be instantiated to perform parallel processing of the partitioned data. The data may be partitioned according to the data partitioning policy and processed according to the processing instructions. And the results may be returned to the application.

1 . A computer system for parallel processing of application data, comprising:
 a data partitioning engine that partitions application data according to a data partitioning policy and processes each of a plurality of data partitions according to processing instructions for the application data;   a data partition processor operably coupled to the data partitioning engine that processes at least one of the plurality of data partitions according to the processing instructions for the application data; and   a storage operably coupled to the data partitioning engine that stores a data partitioning process table with information including an identification of each of the plurality of data partitions and processing status of each of the plurality of data partitions.   
   
   
       2 . The system of  claim 1  further comprising a data partition status monitor operable coupled to the data partitioning engine that monitors and updates the processing status of at least one of the plurality of data partitions. 
   
   
       3 . The system of  claim 1  further comprising an application operably coupled to the data partitioning engine that sends a request to partition the application data and process each of the plurality of data partitions according to the processing instructions for the application data. 
   
   
       4 . The system of  claim 3  further comprising a data processing interface operably coupled the application that receives the request to partition the application data and process each of the plurality of data partitions according to the processing instructions for the application data and sends the request to the data partitioning engine. 
   
   
       5 . The system of  claim 3  further comprising the data partitioning policy operably coupled to the application that specifies instructions for partitioning the application data. 
   
   
       6 . The system of  claim 3  further comprising the processing instructions operably coupled to the application that specifies data processing instructions for processing the application data. 
   
   
       7 . A computer-readable medium having computer-executable components comprising the system of  claim 1 . 
   
   
       8 . A computer-implemented method for parallel processing of application data, comprising:
 receiving a request to perform parallel processing of application data;   partitioning the application data into a plurality of data partitions specified by a data partitioning policy;   processing the plurality of data partitions asynchronously by a plurality of data processors according to processing instructions for the application data; and   outputting results from processing the plurality of data partitions according to the processing instructions for the application data.   
   
   
       9 . The method of  claim 8  further comprising instantiating the plurality of data processors to asynchronously process the plurality of data partitions according to processing instructions for the application data. 
   
   
       10 . The method of  claim 8  further comprising instantiating a plurality of data partition monitors that asynchronously monitor a processing status of each of the plurality of data partitions. 
   
   
       11 . The method of  claim 8  further comprising initializing a processing status of each of the plurality of data partitions. 
   
   
       12 . The method of  claim 8  further comprising monitoring a processing status of each of the plurality of data partitions. 
   
   
       13 . The method of  claim 8  further comprising updating a processing status of each of the plurality of data partitions. 
   
   
       14 . The method of  claim 8  wherein receiving the request to perform parallel processing of application data comprises receiving an address of a data table. 
   
   
       15 . The method of  claim 8  further comprising obtaining the data partitioning policy from the application for partitioning the application data into the plurality of data partitions specified by the data partitioning policy. 
   
   
       16 . The method of  claim 15  further comprising obtaining a number of partitions from the data partitioning policy for partitioning the application data into the plurality of data partitions. 
   
   
       17 . The method of  claim 8  further comprising obtaining the processing instructions for the application data from the application for processing the plurality of data partitions asynchronously by a plurality of data processors. 
   
   
       18 . A computer-readable medium having computer-executable instructions for performing the method of  claim 8 . 
   
   
       19 . A computer system for parallel processing of application data, comprising:
 means for receiving instructions to partition application data into a plurality of data partitions;   means for receiving instructions to process each of the plurality of data partitions;   means for partitioning the application data into the plurality of data partitions;   means for processing each of the plurality of data partitions; and   means for outputting the results of processing each of the plurality of data partitions.   
   
   
       20 . The computer system of  claim 19  further comprising:
 means for sending the instructions to partition the application data into the plurality of data partitions;   means for sending the instructions to process each of the plurality of data partitions.