Data redistribution in shared nothing architecture

Abstract

A system and method for data redistribution. In one embodiment, the method includes dividing data into batches at a sending partition; populating a first data structure with the first pages and the first control information in a first data structure; storing the first data structure in a cache at the sending partition; sending the changes over the network to the receiving partition; receiving a notification that the changes have been successfully stored in the second hard disk at the receiving partition; and storing, in response to the notification, the changes on the first hard disk at the sending partition.

1 . A method comprising:
 dividing data into batches at a sending partition, wherein the data is to be redistributed to a receiving partition, wherein each batch comprises a plurality of first pages and first control information, wherein the plurality of first pages includes changes to a memory, and wherein the control information is used to restart a distribution process in the event of a failure;   populating a first data structure with the first pages and the first control information in a first data structure;   storing the first data structure in a cache at the sending partition, wherein the changes are not stored to a first hard disk at the sending partition until after the changes are successfully stored at the receiving partition;   sending the changes over the network to the receiving partition, wherein the receiving partition populates a second data structure with second pages and second control information, where the plurality of second pages includes the changes, and wherein the changes are subsequently stored in a second hard disk at the receiving partition;   receiving a notification that the changes have been successfully stored in the second hard disk at the receiving partition; and   storing, in response to the notification, the changes on the first hard disk at the sending partition.   
     
     
         2 . The method of  claim 1  further comprising removing the control information from the data structure to free up memory space. 
     
     
         3 . The method of  claim 1  further comprising:
 restarting the populating, storing, and sending steps in response to a failure during redistribution of data; and   causing an undo operation at the receiving partition in response to the failure.   
     
     
         4 . A system comprising:
 a processor; and   a memory for storing an application; and   a first hard disk coupled to the processor, wherein the application is operable to cause the processor to:   divide data into batches at a sending partition, wherein the data is to be redistributed to a receiving partition, wherein each batch comprises a plurality of first pages and first control information, wherein the plurality of first page includes changes to a memory, and wherein the control information is used to restart a distribution process in the event of a failure;   populate a first data structure with the first pages and the first control information in a first data structure;   store the first data structure in a cache at the sending partition, wherein the changes are not stored to the first hard disk at the sending partition until after the changes are successfully stored at the receiving partition;   send the changes over the network to the receiving partition, wherein the receiving partition populates a second data structure with second pages and second control information, where the plurality of second pages includes the changes, and wherein the changes are subsequently stored in a second hard disk at the receiving partition;   receive a notification that the changes have been successfully stored in the second hard disk at the receiving partition; and   store, in response to the notification, the changes on the first hard disk at the sending partition.   
     
     
         5 . The system of  claim 1  wherein the application is operable to cause the processor to:
 restart the populating, storing, and sending steps in response to a failure during redistribution of data; and   cause an undo operation at the receiving partition in response to the failure.