System and method for network interface controller based data deduplication

Abstract

Methods and systems for managing storage of data in a distributed system is disclosed. To manage storage of data in a distributed system, a data processing system may include a network interface controller (NIC). The NIC may present emulated storages that may be used for data storage. The emulated storage devices may utilize storage resources of storage devices. The NIC and the storage devices may implement a distributed deduplication process. The NIC may segment data into chunks and obtain fingerprints of the chunks. The fingerprints may be provided to the storage which may check the fingerprints against fingerprints of already stored chunks. The storage may request the chunks corresponding to the fingerprints that did not match any fingerprints of the already stored chunks. The NIC may provide only those requested chunks to the storage before discarding all of the chunks.

What is claimed is: 
     
       1. A computer-implemented method for managing data storage in a distributed system, the method comprising:
 obtaining, by a Network Interface Controller (NIC) of a data processing system, data for storage; 
 segmenting, by the NIC, the data into chunks; 
 obtaining, by the NIC, fingerprints for the chunks; 
 providing, by the NIC, batches of the fingerprints to a storage; 
 after providing the batches of the fingerprints to the storage, providing, by the NIC and to the storage for storage, a first portion of the chunks corresponding to a first portion of the fingerprints that are new, wherein the first portion of the chunks are provided to the storage in response to receiving instructions from the storage for the first portion of the chunks; and 
 discarding, by the NIC and based on the instructions from the storage, the chunks without providing to the storage a second portion of the chunks corresponding to a second portion of the fingerprints that are not new. 
 
     
     
       2. The computer-implemented method of  claim 1 , wherein the storage is operably connected to the NIC via a network, and the data is obtained from compute resources of the data processing system via a bus of the data processing system. 
     
     
       3. The computer-implemented method of  claim 2 , further comprising:
 presenting an emulated storage device to the compute resources, 
 wherein the data is obtained via communication over the bus, the communication being directed to the emulated storage device. 
 
     
     
       4. The computer-implemented method of  claim 3 , wherein the compute resources believe that the data is stored in a storage directly connected to the bus. 
     
     
       5. The computer-implemented method of  claim 2 , further comprising:
 determining communication characteristics of a connection between the NIC and the storage via the network; and 
 identifying a batch size based on the communication characteristics, 
 wherein providing the batches comprises:
 obtaining a batch of the batches based on the identified batch size. 
 
 
     
     
       6. The computer-implemented method of  claim 5 , wherein obtaining the batch comprises:
 adding a portion of the fingerprints to the batch so that the batch has a size substantially similar to the identified batch size. 
 
     
     
       7. The computer-implemented method of  claim 1 , wherein obtaining the fingerprints comprises:
 obtaining hashes for the chunks, the hashes being used as the fingerprints, and a hash function used to obtain the hashes being substantially collision free. 
 
     
     
       8. The computer-implemented method of  claim 7 , wherein obtaining the hashes for the chunks comprises:
 sending the chunks to a communication security processor of the NIC; and 
 receiving the hashes from the communication security processor. 
 
     
     
       9. The computer-implemented method of  claim 1 , wherein providing the first portion of the chunks comprises, by the NIC:
 receiving, from the storage and in response to the NIC providing the batches of the fingerprints to a storage, the instructions from the storage for the first portion of the chunks, wherein the instructions include a list specifying the first portion of the fingerprints that are new; and 
 using the list to select the first portion of the chunks. 
 
     
     
       10. The computer-implemented method of  claim 1 , wherein the instructions are received from the storage in a response batch that is responsive to a batch of the batches of the fingerprints provided to the storage, and wherein the storage is a network storage device that is remotely connected to the NIC via a network. 
     
     
       11. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations for managing storage of data in a distributed system, the operations comprising:
 obtaining, by a Network Interface Controller (NIC) of a data processing system, data for storage; 
 segmenting, by the NIC, the data into chunks; 
 obtaining, by the NIC, fingerprints for the chunks; 
 providing, by the NIC, batches of the fingerprints to a storage; 
 after providing the batches of the fingerprints to the storage, providing, by the NIC and to the storage for storage, a first portion of the chunks corresponding to a first portion of the fingerprints that are new, wherein the first portion of the chunks are provided to the storage in response to receiving instructions from the storage for the first portion of the chunks; and 
 discarding, by the NIC and based on the instructions from the storage, the chunks without providing to the storage a second portion of the chunks corresponding to a second portion of the fingerprints that are not new. 
 
     
     
       12. The non-transitory machine-readable medium of  claim 11 , wherein the storage is operably connected to the NIC via a network, and the data is obtained from compute resources of the data processing system via a bus of the data processing system. 
     
     
       13. The non-transitory machine-readable medium of  claim 12 , wherein the operations further comprise:
 presenting an emulated storage device to the compute resources, 
 wherein the data is obtained via communication over the bus, the communication being directed to the emulated storage device. 
 
     
     
       14. The non-transitory machine-readable medium of  claim 13 , wherein the compute resources believe that the data is stored in a storage directly connected to the bus. 
     
     
       15. The non-transitory machine-readable medium of  claim 12 , wherein the operations further comprise:
 determining communication characteristics of a connection between the NIC and the storage via the network; and 
 identifying a batch size based on the communication characteristics, 
 wherein providing the batches comprises:
 obtaining a batch of the batches based on the identified batch size. 
 
 
     
     
       16. A data processing system, comprising:
 compute resources; and 
 a Network Interface controller (NIC), comprising:
 a processor, and 
 a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations for managing storage of data in a distributed system, the operations comprising:
 obtaining data for storage; 
 segmenting the data into chunks; 
 obtaining fingerprints for the chunks; 
 providing batches of the fingerprints to a storage; 
 after providing the batches of the fingerprints to the storage, providing, to the storage for storage, a first portion of the chunks corresponding to a first portion of the fingerprints that are new, wherein the first portion of the chunks are provided to the storage in response to receiving instructions from the storage for the first portion of the chunks; and 
 
 discarding, based on the instructions from the storage, the chunks without providing to the storage a second portion of the chunks corresponding to a second portion of the fingerprints that are not new. 
 
 
     
     
       17. The data processing system of  claim 16 , wherein the storage is operably connected to the NIC via a network, and the data is obtained from the compute resources via a bus of the data processing system. 
     
     
       18. The data processing system of  claim 17 , wherein the operations further comprise:
 presenting an emulated storage device to the compute resources, 
 wherein the data is obtained via communication over the bus, the communication being directed to the emulated storage device. 
 
     
     
       19. The data processing system of  claim 18 , wherein the compute resources believe that the data is stored in a storage directly connected to the bus. 
     
     
       20. The data processing system of  claim 17 , wherein the operations further comprise:
 determining communication characteristics of a connection between the NIC and the storage via the network; and 
 identifying a batch size based on the communication characteristics, 
 wherein providing the batches comprises:
 obtaining a batch of the batches based on the identified batch size.