Repartitioning data in a distributed computing system

Abstract

Methods, systems, and apparatus for allocating, by a source of one or more sources, a segment of a data file of a transient memory for exclusive access by the source, the transient memory being a distributed in-memory file system that supports remote direct memory access; writing, by the source, data from an initial partition to one or more blocks within the allocated segment of the data file, wherein a portion of the initial partition is written to a first block of the one or more blocks; publishing, by the source, the segment of the data file of the transient memory to be accessible for reading by one or more sinks; and reading by a particular sink of the one or more sinks, a particular block of the published segment of the data file of the transient memory, wherein the particular block is associated with the particular sink.

What is claimed is: 
     
       1. A computer-implemented method comprising:
 allocating, by a source of one or more sources that each comprise an initial partition of data to be repartitioned, a segment of a transient memory for exclusive access by the source; 
 writing, by the source, data from the initial partition to one or more blocks within the allocated segment of the transient memory, wherein a portion of the initial partition of the data is written to a first block of the one or more blocks within the allocated segment of the transient memory; 
 publishing, by the source, the segment of the transient memory to be accessible for reading by one or more sinks that each comprise a final partition of the data; and 
 reading, by a particular sink of the one or more sinks, a particular block of the published segment of the transient memory, wherein the particular block is associated with the particular sink. 
 
     
     
       2. The method of  claim 1 , wherein the transient memory is a distributed in-memory file system that supports remote direct memory access operations. 
     
     
       3. The method of  claim 1 , wherein the transient memory comprises a marker file that includes metadata related to each segment in the transient memory, wherein the metadata comprises information about segment length and segment commit status, and
 wherein publishing, by the source, the segment of the transient memory to be accessible for reading by the one or more sinks comprises updating the metadata. 
 
     
     
       4. The method of  claim 3 , wherein a failure indicator is set for a particular segment, indicating a failure in writing data to the particular segment, based on metadata included in the marker file. 
     
     
       5. The method of  claim 1 ,
 wherein the transient memory comprises a counter used to coordinate writes to the transient memory and to exclusively allocate segments in the transient memory to each source, and 
 wherein allocating, by a source of the one or more sources, a segment of the transient memory of the transient memory for exclusive access by the source comprises updating the counter. 
 
     
     
       6. The method of  claim 1 ,
 wherein blocks in a published segment of the transient memory are flushed to a permanent memory, 
 wherein reading, by a particular sink of the one or more sinks, a particular block of the published segment of the transient memory comprises reading the particular block from the permanent memory. 
 
     
     
       7. The method of  claim 6 ,
 wherein the size of the transient memory is maintained to satisfy a predetermined threshold level, by flushing blocks in a published segment of the transient memory to the permanent memory, resizing, and trimming the transient memory, 
 wherein flushing, resizing, and trimming occur at predetermined intervals of time. 
 
     
     
       8. A system, comprising:
 one or more processors; and 
 a memory storing instructions that are operable, when executed, to cause the one or more processors to perform operations comprising: 
 allocating, by a source of one or more sources that each comprise an initial partition of data to be repartitioned, a segment of a transient memory for exclusive access by the source; 
 writing, by the source, data from the initial partition to one or more blocks within the allocated segment of the transient memory, wherein a portion of the initial partition of the data is written to a first block of the one or more blocks within the allocated segment of the transient memory; 
 publishing, by the source, the segment of the transient memory to be accessible for reading by one or more sinks that each comprise a final partition of the data; and 
 reading, by a particular sink of the one or more sinks, a particular block of the published segment of the transient memory, wherein the particular block is associated with the particular sink. 
 
     
     
       9. The system of  claim 8 , wherein the transient memory is a distributed in-memory file system that supports remote direct memory access operations. 
     
     
       10. The system of  claim 8 , wherein the transient memory comprises a marker file that includes metadata related to each segment in the transient memory, wherein the metadata comprises information about segment length and segment commit status, and
 wherein publishing, by the source, the segment of the transient memory to be accessible for reading by the one or more sinks comprises updating the metadata. 
 
     
     
       11. The system of  claim 10 , wherein a failure indicator is set for a particular segment, indicating a failure in writing data to the particular segment, based on metadata included in the marker file. 
     
     
       12. The system of  claim 8 ,
 wherein the transient memory comprises a counter used to coordinate writes to the transient memory and to exclusively allocate segments in the transient memory to each source, and 
 wherein allocating, by a source of the one or more sources, a segment of the transient memory of the transient memory for exclusive access by the source comprises updating the counter. 
 
     
     
       13. The system of  claim 8 ,
 wherein blocks in a published segment of the transient memory are flushed to a permanent memory, 
 wherein reading, by a particular sink of the one or more sinks, a particular block of the published segment of the transient memory comprises reading the particular block from the permanent memory. 
 
     
     
       14. The system of  claim 13 ,
 wherein the size of the transient memory is maintained to satisfy a predetermined threshold level, by flushing blocks in a published segment of the transient memory to the permanent memory, resizing, and trimming the transient memory, 
 wherein flushing, resizing, and trimming occur at predetermined intervals of time. 
 
     
     
       15. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:
 allocating, by a source of one or more sources that each comprise an initial partition of data to be repartitioned, a segment of a transient memory for exclusive access by the source; 
 writing, by the source, data from the initial partition to one or more blocks within the allocated segment of the transient memory, wherein a portion of the initial partition of the data is written to a first block of the one or more blocks within the allocated segment of the transient memory; 
 publishing, by the source, the segment of the transient memory to be accessible for reading by one or more sinks that each comprise a final partition of the data; and 
 reading, by a particular sink of the one or more sinks, a particular block of the published segment of the transient memory, wherein the particular block is associated with the particular sink. 
 
     
     
       16. The medium of  claim 15 , wherein the transient memory is a distributed in-memory file system that supports remote direct memory access operations. 
     
     
       17. The medium of  claim 15 , wherein the transient memory comprises a marker file that includes metadata related to each segment in the transient memory, wherein the metadata comprises information about segment length and segment commit status, and
 wherein publishing, by the source, the segment of the transient memory to be accessible for reading by the one or more sinks comprises updating the metadata. 
 
     
     
       18. The medium of  claim 17 , wherein a failure indicator is set for a particular segment, indicating a failure in writing data to the particular segment, based on metadata included in the marker file. 
     
     
       19. The medium of  claim 15 ,
 wherein the transient memory comprises a counter used to coordinate writes to the transient memory and to exclusively allocate segments in the transient memory to each source, and 
 wherein allocating, by a source of the one or more sources, a segment of the transient memory of the transient memory for exclusive access by the source comprises updating the counter. 
 
     
     
       20. The medium of  claim 15 ,
 wherein blocks in a published segment of the transient memory are flushed to a permanent memory, 
 wherein reading, by a particular sink of the one or more sinks, a particular block of the published segment of the transient memory comprises reading the particular block from the permanent memory.