Reducing number of write operations relative to delivery of out-of-order RDMA send messages by managing reference counter

Abstract

An RNIC implementation that performs direct data placement to memory where all segments of a particular connection are aligned, or moves data through reassembly buffers where all segments of a particular connection are non-aligned. The type of connection that cuts-through without accessing the reassembly buffers is referred to as a “Fast” connection because it is highly likely to be aligned, while the other type is referred to as a “Slow” connection. When a consumer establishes a connection, it specifies a connection type. The connection type can change from Fast to Slow and back. The invention reduces memory bandwidth, latency, error recovery using TCP retransmit and provides for a “graceful recovery” from an empty receive queue. The implementation also may conduct CRC validation for a majority of inbound DDP segments in the Fast connection before sending a TCP acknowledgement (Ack) confirming segment reception.

1. A method of reducing a number of write operations relative to delivery of out-of-order RDMA Send messages, the method comprising the steps of:
 providing a reference counter with a completion queue element; 
 setting the reference counter to a number of RDMA Send messages completed for a selected TCP hole; 
 reducing the reference counter by one for each Poll-for-Completion conducted by an RDMA verb interface; and 
 removing the CQE from a respective completion queue where the reference counter becomes zero. 
 
   
   
     2. The method of  claim 1 , further comprising the step of having the completion queue element include at least part of completion data. 
   
   
     3. The method of  claim 2 , wherein a remainder of the completion data is included in a work queue element associated with one or more RDMA Send messages. 
   
   
     4. The method of  claim 2 , further comprising the step of indicating one of: 1) the completion queue element includes all completion data, and 2) completion queue element completion data is at least partially included in a work queue element associated with one or more RDMA Send messages. 
   
   
     5. The method of  claim 1 , further comprising the step of updating a work queue element of a send queue only if it has greater than a threshold number of pending RDMA Send messages. 
   
   
     6. The method of  claim 5 , wherein the threshold is commensurate with resources allocated for storing information for pending RDMA Send messages. 
   
   
     7. The method of  claim 1 , wherein a number of write operations equals N+1, where N is a number of completed RDMA Send messages that were pending prior to the selected TCP hole closing. 
   
   
     8. A computer system for reducing a number of write operations relative to delivery of out-of-order RDMA Send messages, the computer system comprising:
 at least one processing unit; 
 a memory operably associated with the at least one processing unit; and 
 at least one RDMA enabled network interface controller storable in the memory and executable by the at least one processing unit for reducing a number of write operations relative to delivery of out-of-order RDMA Send messages, the tool including:
 means for setting a reference counter of a completion queue element to a number of RDMA Send messages completed for a selected TCP hole; 
 means for reducing the reference counter by one for each Poll-for-Completion conducted by an RDMA verb interface; and 
 means for removing the completion queue element from a respective completion queue in a case that the reference counter becomes zero. 
 
 
   
   
     9. The computer system of  claim 8 , further comprising means for updating a work queue element of a send queue only if it has greater than a threshold number of pending RDMA Send messages. 
   
   
     10. The computer system of  claim 9 , wherein the threshold is commensurate with resources allocated for storing information for pending RDMA Send messages. 
   
   
     11. The computer system of  claim 8 , further comprising means for storing at least part of completion data in the completion queue element and a remainder of the completion data in a work queue element associated with one or more RDMA Send messages. 
   
   
     12. The computer system of  claim 11 , further comprising means for indicating one of: 1) the completion queue element includes all completion data, and 2) completion queue element completion data is at least partially included in a work queue element associated with one or more RDMA Send messages. 
   
   
     13. The computer system of  claim 8 , wherein a number of write operations equals N+1, where N is a number of completed RDMA Send messages that were pending prior to the selected TCP hole closing. 
   
   
     14. A computer program product comprising a computer useable storage medium having computer readable program code embodied therein for reducing a number of write operations relative to delivery of out-of-order RDMA Send messages, the program product comprising:
 program code configured to set a reference counter associated with a completion queue element to a number of RDMA Send messages completed for a selected TCP hole; 
 program code configured to reduce the reference counter by one for each Poll-for-Completion conducted by an RDMA verb interface; and 
 program code configured to remove the completion queue element from a respective completion queue in a case that the reference counter becomes zero. 
 
   
   
     15. The program product of  claim 14 , further comprising program code configured to store at least part of completion data in the completion queue element. 
   
   
     16. The program product of  claim 15 , wherein the storing program code stores a remainder of the completion data in a work queue element associated with one or more RDMA Send messages. 
   
   
     17. The program product of  claim 15 , further comprising an indicator of one of: 1) the completion queue element includes all completion data, and 2) completion queue element completion data is at least partially included in a work queue element associated with one or more RDMA Send messages. 
   
   
     18. The program product of  claim 15 , wherein a number of write operations equals N+1, where N is a number of completed RDMA Send messages that were pending prior to the selected TCP hole closing. 
   
   
     19. The program product of  claim 14 , further comprising program code configured to update a work queue element of a send queue only if it has greater than a threshold number of pending RDMA Send messages. 
   
   
     20. The program product of  claim 14 , wherein the threshold is commensurate with resources allocated for storing information for pending RDMA Send messages.