Remote Direct Memory Access ('RDMA') In A Parallel Computer

Abstract

Remote direct memory access (‘RDMA’) in a parallel computer, the parallel computer including a plurality of nodes, each node including a messaging unit, including: receiving an RDMA read operation request that includes a virtual address representing a memory region at which to receive data to be transferred from a second node to the first node; responsive to the RDMA read operation request: translating the virtual address to a physical address; creating a local RDMA object that includes a counter set to the size of the memory region; sending a message that includes an DMA write operation request, the physical address of the memory region on the first node, the physical address of the local RDMA object on the first node, and a remote virtual address on the second node; and receiving the data to be transferred from the second node.

1 . A method of remote direct memory access (‘RDMA’) in a parallel computer, the parallel computer including a plurality of nodes, each node including a messaging unit, the method comprising:
 receiving, by a kernel of the first node through an application programming interface (‘API’), an RDMA read operation request that includes a virtual address representing a memory region at which to receive data to be transferred from a second node to the first node; 
 responsive to the RDMA read operation request: 
 translating, by the kernel of the first node, the virtual address to a physical address; 
 creating, by the kernel of the first node, a local RDMA object that includes a counter set to the size of the memory region; 
 sending, from a messaging unit in the first node to a messaging unit in a second node, a message that includes an DMA write operation request, the physical address of the memory region on the first node, the physical address of the local RDMA object on the first node, and a remote virtual address on the second node representing the data to be transferred from the second node to the first node; and 
 receiving, by the first node responsive to the second node's execution of the DMA write operation request, the data to be transferred from the second node. 
 
     
     
         2 . The method of  claim 1  further comprising:
 receiving, by the messaging unit in the second node, the message; 
 translating, by a kernel on the second node, the remote virtual address on the second node to a physical address on the second node; and 
 transferring, by one or more DMA operations executing on the messaging unit in the second node, the data to be transferred from the second node to the first node. 
 
     
     
         3 . The method of  claim 1  further comprising:
 decrementing, by the kernel on the first node, the counter by an amount equal to the size of data received from the second node; and 
 notifying, by the kernel on the first node, a user-level application that the data to be transferred from the second node to the first node has been received when the counter is equal to zero. 
 
     
     
         4 . The method of  claim 3  wherein notifying, by the kernel on the first node, a user-level application that the data to be transferred from the second node to the first node has been received includes raising an interrupt in the user-level application. 
     
     
         5 . The method of  claim 3  wherein notifying, by the kernel on the first node, a user-level application that the data to be transferred from the second node to the first node has been received includes being polled by the user-level application. 
     
     
         6 . The method of  claim 1  further comprising sending an error message indicating that the RDMA read operation request cannot be serviced. 
     
     
         7 . A method of remote direct memory access (‘RDMA’) in a parallel computer, the parallel computer including a plurality of nodes, each node including a messaging unit, the method comprising:
 receiving, by a kernel of a first node through an application programming interface (‘API’), an RDMA write operation request that includes a virtual address representing a memory region in the first node from which to send data to a second node; 
 responsive to the RDMA write operation request: 
 translating, by the kernel of the first node, the virtual address to a physical address; 
 creating, by the kernel of the first node, a local RDMA object that includes a counter set to the size of the memory region; 
 sending, from a messaging unit in the first node to a messaging unit in a second node, a message that includes an DMA read operation request, the physical address of the memory region on the first node, the physical address of the local RDMA object on the first node, and a remote virtual address on the second node representing a memory region in the second node for receiving data from the first node; and 
 sending, from the first node responsive to the second node's execution of the DMA read operation request, the data to be received by the second node. 
 
     
     
         8 . The method of  claim 7  further comprising:
 receiving, by the messaging unit in the second node, the message; 
 translating, by a kernel on the second node, the remote virtual address on the second node to a physical address on the second node; and 
 receiving, by one or more DMA operations executing on the messaging unit in the second node, the data to be transferred the first node to the second node. 
 
     
     
         9 . The method of  claim 7  further comprising:
 decrementing, by the kernel on the first node, the counter by an amount equal to the size of data sent to the second node; and 
 notifying, by the kernel on the first node, a user-level application that the data to be transferred to the second node has been transferred when the counter is equal to zero. 
 
     
     
         10 . The method of  claim 9  wherein notifying, by the kernel on the first node, a user-level application that the data to be transferred to the second node has been transferred includes raising an interrupt in the user-level application. 
     
     
         11 . The method of  claim 9  wherein notifying, by the kernel on the first node, a user-level application that the data to be transferred to the second node has been transferred includes being polled by the user-level application. 
     
     
         12 . The method of  claim 7  further comprising sending an error message to a user-level application indicating that the RDMA write operation request cannot be serviced. 
     
     
         13 . An apparatus for remote direct memory access (‘RDMA’) in a parallel computer, the parallel computer including a plurality of nodes, each node including a messaging unit, the apparatus comprising a computer processor, a computer memory operatively coupled to the computer processor, the computer memory having disposed within it computer program instructions that, when executed by the processor, cause the apparatus to carry out the steps of:
 receiving, by a kernel of the first node through an application programming interface (‘API’), an RDMA read operation request that includes a virtual address representing a memory region at which to receive data to be transferred from a second node to the first node; 
 responsive to the RDMA read operation request: 
 translating, by the kernel of the first node, the virtual address to a physical address; 
 creating, by the kernel of the first node, a local RDMA object that includes a counter set to the size of the memory region; 
 sending, from a messaging unit in the first node to a messaging unit in a second node, a message that includes an DMA write operation request, the physical address of the memory region on the first node, the physical address of the local RDMA object on the first node, and a remote virtual address on the second node representing the data to be transferred from the second node to the first node; and 
 receiving, by the first node responsive to the second node's execution of the DMA write operation request, the data to be transferred from the second node. 
 
     
     
         14 . The apparatus of  claim 13  further comprising computer program instructions that, when executed by the processor, cause the apparatus to carry out the steps of:
 receiving, by the messaging unit in the second node, the message; 
 translating, by a kernel on the second node, the remote virtual address on the second node to a physical address on the second node; and 
 transferring, by one or more DMA operations executing on the messaging unit in the second node, the data to be transferred from the second node to the first node. 
 
     
     
         15 . The apparatus of  claim 13  further comprising computer program instructions that, when executed by the processor, cause the apparatus to carry out the steps of:
 decrementing, by the kernel on the first node, the counter by an amount equal to the size of data received from the second node; and 
 notifying, by the kernel on the first node, a user-level application that the data to be transferred from the second node to the first node has been received when the counter is equal to zero. 
 
     
     
         16 . The apparatus of  claim 15  wherein notifying, by the kernel on the first node, a user-level application that the data to be transferred from the second node to the first node has been received includes raising an interrupt in the user-level application. 
     
     
         17 . The apparatus of  claim 15  wherein notifying, by the kernel on the first node, a user-level application that the data to be transferred from the second node to the first node has been received includes being polled by the user-level application. 
     
     
         18 . The apparatus of  claim 13  further comprising computer program instructions that, when executed by the processor, cause the apparatus to carry out the step of sending an error message indicating that the RDMA read operation request cannot be serviced. 
     
     
         19 . An apparatus for remote direct memory access (‘RDMA’) in a parallel computer, the parallel computer including a plurality of nodes, each node including a messaging unit, the apparatus comprising a computer processor, a computer memory operatively coupled to the computer processor, the computer memory having disposed within it computer program instructions that, when executed by the processor, cause the apparatus to carry out the steps of:
 receiving, by a kernel of a first node through an application programming interface (‘API’), an RDMA write operation request that includes a virtual address representing a memory region in the first node from which to send data to a second node; 
 responsive to the RDMA write operation request: 
 translating, by the kernel of the first node, the virtual address to a physical address; 
 creating, by the kernel of the first node, a local RDMA object that includes a counter set to the size of the memory region; 
 sending, from a messaging unit in the first node to a messaging unit in a second node, a message that includes an DMA read operation request, the physical address of the memory region on the first node, the physical address of the local RDMA object on the first node, and a remote virtual address on the second node representing a memory region in the second node for receiving data from the first node; and 
 sending, from the first node responsive to the second node's execution of the DMA read operation request, the data to be received by the second node. 
 
     
     
         20 . The apparatus of  claim 19  further comprising computer program instructions that, when executed by the processor, cause the apparatus to carry out the steps of:
 receiving, by the messaging unit in the second node, the message; 
 translating, by a kernel on the second node, the remote virtual address on the second node to a physical address on the second node; and 
 receiving, by one or more DMA operations executing on the messaging unit in the second node, the data to be transferred the first node to the second node. 
 
     
     
         21 . The apparatus of  claim 19  further comprising computer program instructions that, when executed by the processor, cause the apparatus to carry out the steps of:
 decrementing, by the kernel on the first node, the counter by an amount equal to the size of data sent to the second node; and 
 notifying, by the kernel on the first node, a user-level application that the data to be transferred to the second node has been transferred when the counter is equal to zero. 
 
     
     
         22 . The apparatus of  claim 21  wherein notifying, by the kernel on the first node, a user-level application that the data to be transferred to the second node has been transferred includes raising an interrupt in the user-level application. 
     
     
         23 . The apparatus of  claim 21  wherein notifying, by the kernel on the first node, a user-level application that the data to be transferred to the second node has been transferred includes being polled by the user-level application. 
     
     
         24 . The apparatus of  claim 19  further comprising computer program instructions that, when executed by the processor, cause the apparatus to carry out the step of sending an error message indicating that the RDMA write operation request cannot be serviced.