Batch processing and data synchronization in cloud-based systems

Abstract

Disclosed are methods and apparatus for efficiently storing application data received from clients in a database stored on a server, such as a cloud-based database. The methods include receiving a batch of a plurality of web operations that specify actions to be performed on data objects that represent application data stored in a cloud database, translating the web operations to a batch of data storage operation sets, creating a temporary database having a subset of contents of the cloud database, applying the data storage operation sets to the temporary database, recording database operations generated by the temporary database based on the plurality of data storage operation sets, and applying the plurality of database operations to the cloud database in a transaction. Translating the web operations can include mapping the actions to be performed on the data objects to transactions to be performed on the cloud database.

What is claimed is: 
     
       1. A method, comprising:
 by a computer device:
 receiving a batch of web operations that specify an action to be performed on application data stored in a cloud database, wherein the batch of web operations comprises one or more constraints associated with a database state; 
 generating a temporary database in response to receiving the batch of web operations, wherein the temporary database includes data copied from the cloud database; 
 removing the one or more constraints from the batch of web operations to yield a modified batch of web operations; 
 translating the modified batch of web operations to a first set of database operations that are compatible with a data storage interface of the cloud database; 
 applying the first set of database operations to the temporary database; 
 generating, in response to applying the first set of database operations to the temporary database, a second set of database operations based on the first set of database operations applied to the temporary database; 
 re-applying the one or more constraints to the second set of database operations; and 
 applying the second set of database operations with the one or more constraints to the cloud database. 
 
 
     
     
       2. The method of  claim 1 , wherein translating the batch of web operations includes:
 mapping the action to be performed on the cloud database, and 
 mapping one or more data objects to one or more application data items in the cloud database. 
 
     
     
       3. The method of  claim 1 , wherein the data copied from the cloud database for the temporary database is limited to data needed for performing the batch of web operations. 
     
     
       4. The method of  claim 1 , wherein the second set of database operations specify one or more relational database actions to be performed on the application data stored in the cloud database. 
     
     
       5. The method of  claim 4 , wherein the second set of database operations comprise operations to insert, update, or delete data in the cloud database. 
     
     
       6. The method of  claim 1 , further comprising:
 determining a sub-batch size based upon available resources; and 
 dividing the batch of web operations into sub-batches having sizes based upon the sub-batch size, wherein translating the batch of web operations comprises translating one sub-batch of the sub-batches to a subset of data storage operations, and applying the first set of database operations to the temporary database comprises applying the subset of data storage operations to the temporary database. 
 
     
     
       7. The method of  claim 1 , wherein the batch of web operations specify the action to be performed on one or more data objects that are identified by one or more web addresses. 
     
     
       8. The method of  claim 7 , wherein the batch of web operations comprise operations to get, insert, update, or delete data associated with the one or more data objects specified in the batch of web operations. 
     
     
       9. The method of  claim 1 , wherein generating the second set of database operations comprises rejecting one or more web operations from the batch of web operations that do not satisfy one or more other constraints. 
     
     
       10. The method of  claim 1 , wherein applying the first set of database operations to the temporary database causes a fault to occur when the temporary database does not contain one or more data items referenced by the first set of database operations. 
     
     
       11. The method of  claim 1 , further comprising:
 after re-applying the one or more constraints to the second set of database operations, removing at least one redundant database operation from the second set of database operations to yield a modified second set of database operations; 
 applying the modified second set of database operations to the cloud database; and 
 deleting the temporary database subsequent to applying the modified second set of database operations to the cloud database. 
 
     
     
       12. The method of  claim 1 , wherein creating the temporary database comprises:
 creating an empty database instance, and 
 storing a predetermined subset of contents of the cloud database in the temporary database. 
 
     
     
       13. The method of  claim 1 , wherein each web operation of the batch of web operations is associated with one or more other constraints that specify conditions in which the web operation is to be performed, and the one or more other constraints are translated to one or more database constraints that are applied to the cloud database. 
     
     
       14. A system, comprising:
 a processor; and 
 a memory configured to store instructions that, when executed by the processor, cause the system to perform steps that include:
 receiving a batch of web operations that specify an action be performed on application data stored in a cloud database, wherein the batch of web operations comprises one or more constraints associated with a database state; 
 generating a temporary database in response to receiving the batch of web operations, wherein the temporary database includes data copied from the cloud database; 
 removing the one or more constraints from the batch of web operations to yield a modified batch of web operations; 
 translating the modified batch of web operations to a first set of database operations that are compatible with a data storage interface of the cloud database; 
 applying the first set of database operations to the temporary database; 
 generating, in response to applying the first set of database operations to the temporary database, a second set of database operations based on the first set of database operations applied to the temporary database; 
 re-applying the one or more constraints to the second set of database operations; and 
 applying the second set of database operations with the one or more constraints to the cloud database. 
 
 
     
     
       15. The system of  claim 14 , wherein the steps further include:
 mapping the action to be performed on the on the cloud database, and 
 mapping one or more data objects to one or more application data items in the cloud database. 
 
     
     
       16. The system of  claim 14 , wherein the steps further include:
 causing a fault to occur when the temporary database does not contain one or more data items referenced by the first set of database operations; and 
 loading the one or more data items into the temporary database in response to the fault. 
 
     
     
       17. The system of  claim 14 , wherein each web operation of the batch of web operations is associated with one or more other constraints that specify conditions in which the web operation is to be performed, and the one or more other constraints are translated to one or more database constraints that are applied to the cloud database. 
     
     
       18. A non-transitory computer readable storage medium configured to store instructions that, when executed by a processor included in a computing device, cause the computing device to carry out steps that include:
 receiving a batch of web operations that specify an action to be performed on application data stored in a cloud database, wherein the batch of web operations comprises one or more constraints associated with a database state; 
 generating a temporary database in response to receiving the batch of web operations, wherein the temporary database includes data copied from the cloud database; 
 removing the one or more constraints from the batch of web operations to yield a modified batch of web operations; 
 translating the modified batch of web operations to a first set of database operations that are compatible with a data storage interface of the cloud database; 
 applying the first set of database operations to the temporary database; 
 generating, in response to applying the first set of database operations to the temporary database, a second set of database operations based on the first set of database operations applied to the temporary database; 
 re-applying the one or more constraints to the second set of database operations; and 
 applying the second set of database operations with the one or more constraints to the cloud database. 
 
     
     
       19. The non-transitory computer readable storage medium of  claim 18 , wherein the data copied from the cloud database for the temporary database is limited to data needed for performing the batch of web operations. 
     
     
       20. The non-transitory computer readable storage medium of  claim 18 , wherein the steps further include:
 determining a sub-batch size based upon available resources; and 
 partitioning the batch of web operations into sub-batches having sizes based upon the sub-batch size, wherein translating the batch of web operations comprises translating one sub-batch of the sub-batches to a subset of data storage operations, and applying the first set of database operations to the temporary database comprises applying the subset of data storage operations to the temporary database.