Data transfers in columnar data systems

Abstract

A computer-implemented method includes receiving a request to transmit column group data to a target node, the column group data comprising C columns within a column-oriented data table and determining a transmission row count R for transmitting the column group data to the target node. The method may also include transmitting a transmission packet comprising R sequentially-ordered data elements for each of the C columns to the target node. The R data elements for each column may be sequentially retrieved from memory. A corresponding method includes receiving, at a target node, a request to receive the column group data, determining the transmission row count R for receiving the column group data, and receiving a transmission packet comprising R sequentially-ordered data elements for each of the C columns and storing the sequentially-ordered data elements within memory. A corresponding computer system and corresponding computer program products are also disclosed herein.

What is claimed is: 
     
       1. A method, executed by one or more processors, the method comprising:
 receiving a request to transmit column group data to a target node, the column group data comprising C selected columns within a column-oriented data table where C is greater than 1 and less than the number of columns in the column-oriented data table; 
 determining an optimal packet size or packet size range for transmitting data to the target node; 
 determining a transmission row count R, corresponding to the optimal packet size or packet size range, for transmitting the column group data to the target node where R is greater than 1; and 
 transmitting a transmission packet comprising R sequentially-ordered data elements for each of the C selected columns to the target node. 
 
     
     
       2. The method of  claim 1 , wherein the R sequentially-ordered data elements for each column reside sequentially within memory. 
     
     
       3. The method of  claim 1 , wherein R is selected to achieve one or more of preventing stalling, supporting serialization, increasing transmission throughput, and decreasing transmission packet latency. 
     
     
       4. The method of  claim 1 , wherein transmitting the transmission packet to the target node comprises sequentially accessing R data elements for each of the C selected columns. 
     
     
       5. The method of  claim 1 , further comprising repeatedly transmitting transmission packets comprising R data elements for each of the C selected columns until all of the column group data is transmitted to the target node. 
     
     
       6. The method of  claim 5 , wherein repeatedly transmitting transmission packets comprises transmitting multiple windowed transmission packets before a packet acknowledgement is received for a previous transmission packet of the multiple windowed transmission packets. 
     
     
       7. The method of  claim 6 , wherein a maximum windowed packet count for windowed transmission packets corresponds to a transmission buffer size or a receiving buffer size. 
     
     
       8. The method of  claim 1 , wherein the data elements within the column-oriented data table are compressed data elements. 
     
     
       9. The method of  claim 1 , wherein transmitting the transmission packet to the target node comprises assembling the transmission packet within a transmission buffer. 
     
     
       10. A method, executed by one or more processors, the method comprising:
 receiving, at a target node, a request to receive column group data, the column group data comprising C selected columns within a column-oriented data table where C is greater than 1 and less than the number of columns in the column-oriented data table; 
 determining an optimal packet size or packet size range for transmitting data to the target node from a source node; 
 determining a transmission row count R, corresponding to the optimal packet size or packet size range, for receiving the column group data where R is greater than 1; and 
 receiving a transmission packet comprising R sequentially-ordered data elements for each of the C selected columns. 
 
     
     
       11. The method of  claim 10 , further comprising processing data from one or more previously received transmission packets concurrent with receiving the transmission packet. 
     
     
       12. The method of  claim 10 , further comprising repeatedly receiving transmission packets comprising R data elements for each of the C selected columns until all of the column group data is received. 
     
     
       13. The method of  claim 10 , wherein receiving the transmission packet comprises sequentially storing R data elements for each of the C selected columns. 
     
     
       14. The method of  claim 10 , further comprising assembling transmission packets received from multiple transmitting nodes into a processing group. 
     
     
       15. The method of  claim 10 , wherein R is selected to achieve one or more of preventing stalling, supporting serialization, maximizing transmission throughput, and minimizing transmission packet latency. 
     
     
       16. The method of  claim 10 , wherein the R data elements for a column reside within memory. 
     
     
       17. A column-oriented data processing system comprising:
 a source node and a target node; 
 the source node configured to receive a request to transmit column group data to the target node, the column group data comprising C selected columns within a column-oriented data table where C is greater than 1 and less than the number of columns in the column-oriented data table; 
 determining an optimal packet size or packet size range for transmitting data to the target node from the source node; 
 the source node further configured to determine a transmission row count corresponding to the optimal packet size or packet size range, for transmitting the column group data to the target node where R is greater than 1; and 
 the source node further configured to transmit a transmission packet comprising R sequentially-ordered data elements for each of the C selected columns to the target node. 
 
     
     
       18. The column-oriented data processing system of  claim 17 , wherein the target node is configured to process data from one or more previously received transmission packets concurrent with receiving the transmission packet. 
     
     
       19. The column-oriented data processing system of  claim 17 , wherein the source node is configured to store elements for each column sequentially within memory. 
     
     
       20. The column-oriented data processing system of  claim 19 , wherein the source node is configured to transmit the transmission packet to the target node by sequentially retrieving R data elements from the memory for each of the C selected columns.