High speed data transfer mechanism

Abstract

A storage and data management system establishes a data transfer pipeline between an application and a storage media using a source data mover and a destination data mover. The data movers are modular software entities which compartmentalize the differences between operating systems and media types. In addition, they independently interact to perform encryption, compression, etc., based on the content of a file as it is being communicated through the pipeline. Headers and chunking of data occurs when beneficial without the application ever having to be aware. Faster access times and storage mapping offer enhanced user interaction.

1. A method for copying data from a source to a destination using a data pipe, the method comprising:
 identifying at least a first characteristic of a first portion of the data; 
 identifying at least a second characteristic of a second portion of the data; 
 copying the first portion of the data through the data pipe in a first chunk in a first format based on the first characteristic; 
 generating a first header describing the contents of the first chunk, the first header including information relating to at least a first storage operation to be performed on the first chunk; 
 copying the second portion of the data through the data pipe in a second chunk in a second format based on the second characteristic, the second format being distinct from the first format; and 
 generating a second header describing the contents of the second chunk, the second header including information relating to at least a second storage operation to be performed on the second chunk. 
 
     
     
       2. The method as recited in  claim 1 , further comprising:
 sending the first chunk with the first header to a first destination; and 
 sending the second chunk with the second header to a second destination. 
 
     
     
       3. The method as recited in  claim 2 , wherein:
 the first header indicates where to store the first chunk; and 
 the second header indicates where to store the second chunk. 
 
     
     
       4. The method as recited in  claim 2  further comprising:
 creating a first data pipe for sending the first chunk to the first destination; and 
 creating a second data pipe for sending the second chunk to the second destination. 
 
     
     
       5. The method as recited in  claim 4  wherein the first data pipe is created based at least on the first characteristic; and the second data pipe is created based at least on the second characteristic. 
     
     
       6. The method as recited in  claim 4  wherein the first data pipe and the second data pipe are created based at least on the respective destination. 
     
     
       7. The method as recited in  claim 3 , further comprising:
 storing the first chunk in a first storage medium in the first format; and 
 storing the second chunk in a second storage medium in the second format, the second storage medium being distinct from the first storage medium. 
 
     
     
       8. The method as recited in  claim 1 , further comprising:
 performing the first storage operation on the first chunk; and 
 performing the second storage operation on the second chunk. 
 
     
     
       9. The method as recited in  claim 8  wherein the first storage operation and the second storage operation comprise a back up operation. 
     
     
       10. The method as recited in  claim 8  wherein the first storage operation and the second storage operation comprise an archive operation. 
     
     
       11. The method as recited in  claim 1  wherein the first characteristic comprises a data type. 
     
     
       12. A method for copying data from a first storage device to a second storage device, the method comprising:
 obtaining data from a first storage device; 
 parsing the data into at least a first chunk and a second chunk, said parsing being based on a first characteristic of the data in the first chunk and a second characteristic of the data in the second chunk; 
 generating a first header for the first chunk, the first header being indicative of a first storage operation to be performed on the first chunk; 
 generating a second header for the second chunk, the second header being indicative of a second storage operation to be performed on the second chunk; 
 sending the first header and the first chunk in a first format over a network to a first storage destination; and 
 sending the second header and the second chunk in a second format over the network to a second storage destination. 
 
     
     
       13. The method of  claim 12 , additionally comprising:
 storing the first chunk in the first format in the first storage destination; and 
 storing the second chunk in the second format in the second storage destination, wherein the first format is different than the second format. 
 
     
     
       14. The method of  claim 13 , wherein the first storage destination is on a first storage medium and the second storage destination is on a second storage medium different from the first storage medium. 
     
     
       15. The method of  claim 12 , wherein the first storage operation comprises a back up operation or an archive operation. 
     
     
       16. The method of  claim 15 , wherein the second storage operation is different than the first storage operation. 
     
     
       17. A system for transferring data between a source storage device and multiple destination storage devices, the system comprising:
 a source storage device configured to store a piece of data; 
 a plurality of destination storage devices; 
 a first data mover module comprising:
 a parsing module configured to divide the piece of data into a plurality of chunks based on different characteristics of the data, 
 a header module configured to generate and associate a header with each of the plurality of chunks, each header comprising information regarding a storage operation to be performed on the associated chunk, and 
 a interface module configured to transmit the plurality of chunks over a network; and 
 
 a second data mover module coupled to the network and configured to receive the plurality of chunks from the network, the second data mover module being further configured to send the plurality of chunks to the plurality of destination storage devices based on the storage operation information stored in each header. 
 
     
     
       18. The system of  claim 17 , wherein the first data mover module is configured to dynamically establish a plurality of data pipes with the second data mover module, each of the plurality of data pipes being configured to transfer at least one of the plurality of chunks between the first data mover module and the second data mover module. 
     
     
       19. The system of  claim 17 , wherein the interface module comprises a plurality of network agents that are dynamically assignable by the first data mover module to transfer the plurality of chunks based on an available bandwidth of the network. 
     
     
       20. The system of  claim 19 , wherein the plurality of destination storage devices comprises different types of storage media.