Storage of Data in a Distributed Storage System

Abstract

A distributed storage system stores data for files. A first blob (binary large object) of data is received. The first blob is split into one or more first chunks of data. Content fingerprints for the first chunks of data are computed. The first chunks of data are stored in a chunk store while and their content fingerprints are stored in a store distinct from the chunk store. A second blob of data is received. The second blob is split into one or more second chunks of data. Content fingerprints for the second chunks of data are computed. Then for a second chunk of data whose content fingerprint matches a content fingerprint of a first chunk of data, a second reference to the corresponding first chunk of data that has a matching content fingerprint is stored, but the second chunk of data is not stored.

1 . A method of storing data for files, implemented at a plurality of instances of a distributed storage system, each instance comprising one or more servers having memory and one or more processors storing one or more programs configured for execution by the one or more processors, the method comprising:
 receiving a first representation of a blob of data having a specified first representation type;   storing the first representation of the blob of data;   creating a metadata record for the blob of data, wherein the metadata record stores metadata for the blob of data, including a name of the blob, the representation type, and a storage location for the first representation of the blob;   receiving a request from a first user at a first client device to create a second representation of the blob with a second representation type distinct from the first representation type, wherein the first client device is distinct from the one or more servers;   creating the requested second representation of the blob having the second representation type;   storing the second representation of the blob of data;   updating the metadata record for the blob of data to indicate the presence of the second representation of the blob with the second representation type;   receiving a request from a second user at a second client device to transfer a copy of the blob to the second client device, wherein the request includes a specified representation type;   based on compared efficiencies, either retrieving the specified representation of the blob from the first instance or deriving and retrieving the specified representation of the blob from the second instance; and   sending the retrieved representation of the blob to the second client device.   
     
     
         2 . The method of  claim 1 , wherein the specified first representation type is explicitly provided. 
     
     
         3 . The method of  claim 1 , wherein the specified first representation type is assigned by default when the first representation type is not explicitly provided. 
     
     
         4 . The method of  claim 1 , wherein the request to create a second representation of the blob comprises a remote procedure call. 
     
     
         5 . The method of  claim 1 , wherein the first representation of the blob is stored in a first chunk store, the second representation of the blob is stored in a second chunk store, and the second chunk store is distinct from the first chunk store. 
     
     
         6 . The method of  claim 1 , wherein the request to transfer a copy of the blob is included with the request to create the second representation of the blob. 
     
     
         7 . The method of  claim 1 , wherein the second user is the same as the first user and the second client device is the same as the first client device. 
     
     
         8 . A distributed storage system having a plurality of instances, each instance comprising one or more servers having:
 memory;   one or more processors; and   one or more programs stored in the memory and configured for execution by the one or more processors, the one or more programs including instructions for:   receiving a first representation of a blob of data having a specified first representation type;   storing the first representation of the blob of data;   creating a metadata record for the blob of data, wherein the metadata record stores metadata for the blob of data, including a name of the blob, the representation type, and a storage location for the first representation of the blob;   receiving a request from a first user at a first client device to create a second representation of the blob with a second representation type distinct from the first representation type, wherein the first client device is distinct from the computer system;   creating the requested second representation of the blob having the second representation type;   storing the second representation of the blob of data;   updating the metadata record for the blob of data to indicate the presence of the second representation of the blob with the second representation type;   receiving a request from a second user at a second client device to transfer a copy of the blob to the second client device, wherein the request includes a specified representation type;   based on compared efficiencies, either retrieving the specified representation of the blob from the first instance or deriving and retrieving the specified representation of the blob from the second instance; and   sending the retrieved representation of the blob to the second client device.   
     
     
         9 . The computer system of  claim 8 , wherein the specified first representation type is explicitly provided. 
     
     
         10 . The computer system of  claim 8 , wherein the specified first representation type is assigned by default when the first representation type is not explicitly provided. 
     
     
         11 . The computer system of  claim 8 , wherein the request to create a second representation of the blob comprises a remote procedure call. 
     
     
         12 . The computer system of  claim 8 , wherein the first representation of the blob is stored in a first chunk store, the second representation of the blob is stored in a second chunk store, and the second chunk store is distinct from the first chunk store. 
     
     
         13 . The computer system of  claim 8 , wherein the request to transfer a copy of the blob is included with the request to create the second representation of the blob. 
     
     
         14 . A non-transitory computer readable storage medium storing one or more programs configured for execution by a distributed storage system having a plurality of instances, each instance comprising one or more servers, the one or more programs comprising instructions for:
 receiving a first representation of a blob of data having a specified first representation type;   storing the first representation of the blob of data;   creating a metadata record for the blob of data, wherein the metadata record stores metadata for the blob of data, including a name of the blob, the representation type, and a storage location for the first representation of the blob;   receiving a request from a first user at a first client device to create a second representation of the blob with a second representation type distinct from the first representation type, wherein the first client device is distinct from the computer system;   creating the requested second representation of the blob having the second representation type;   storing the second representation of the blob of data;   updating the metadata record for the blob of data to indicate the presence of the second representation of the blob with the second representation type;   receiving a request from a second user at a second client device to transfer a copy of the blob to the second client device, wherein the request includes a specified representation type;   based on compared efficiencies, either retrieving the specified representation of the blob from the first instance or deriving and retrieving the specified representation of the blob from the second instance; and   sending the retrieved representation of the blob to the second client device.   
     
     
         15 . The non-transitory computer readable storage of  claim 14 , wherein the specified first representation type is explicitly provided. 
     
     
         16 . The non-transitory computer readable storage of  claim 14 , wherein the specified first representation type is assigned by default when the first representation type is not explicitly provided. 
     
     
         17 . The non-transitory computer readable storage of  claim 14 , wherein the request to create a second representation of the blob comprises a remote procedure call. 
     
     
         18 . The non-transitory computer readable storage of  claim 14 , wherein the first representation of the blob is stored in a first chunk store, the second representation of the blob is stored in a second chunk store, and the second chunk store is distinct from the first chunk store. 
     
     
         19 . The non-transitory computer readable storage of  claim 14 , wherein the request to transfer a copy of the blob is included with the request to create the second representation of the blob.