Method for handling large object files in an object storage system

Abstract

Several different embodiments of a segmented object storage system are described. The object storage system divides files into a number of object segments, each segment corresponding to a portion of the object, and stores each segment individually in the cloud storage system. The system also generates and stores a manifest file describing the relationship of the various segments to the original data file. Requests to retrieve the segmented file are fulfilled by consulting the manifest file and using the information from the manifest to reconstitute the original data file from the constituent segments. Modifying, appending to, or truncating the object is accomplished by manipulating individual segments and the manifest file. In further embodiments, manipulation of the individual object segments and/or the manifest is used to implement copy-on-write, snapshotting, software transactional memory, and peer-to-peer transmission of the large file.

1 . (canceled) 
     
     
         2 . A method for providing object data from a cloud storage system, the method comprising:
 storing each of a plurality of computer readable information object segments individually, wherein each of the plurality of computer readable information object segments correspond to a portion of the data from a first data source;   receiving a computer readable request to download object data corresponding to one or more of the computer readable information object segments;   determining whether the object data corresponds to all the data from the first data source;   receiving a manifest file describing the relationships of the computer readable information object segments to the first data source;   reconstituting the requested object data from the plurality of computer readable information object segments; and   responding to the request with the reconstituted object data from the one or more of the computer readable information object segments.   
     
     
         3 . The method of  claim 2 , further comprising:
 performing a checksum on the reconstituted object data; and   comparing the result of the checksum against data verification results included in the manifest file, prior to responding to the request with the reconstituted object data.   
     
     
         4 . The method of  claim 2 , wherein reconstituting the requested object data from the plurality of object segments further includes retrieving the plurality of computer readable information object segments from one or more locations in a file storage system identified by the manifest file. 
     
     
         5 . The method of  claim 2 , wherein reconstituting the requested object data from the plurality of object segments further includes placing the plurality of computer readable information object segments in an order identified by the manifest file. 
     
     
         6 . The method of  claim 2 , wherein reconstituting the requested object data from the plurality of object segments further includes performing an error correction operation for one or more of the plurality of computer readable information object segments, using error correction information stored in the manifest file. 
     
     
         7 . A method for storing object data in a cloud storage system, the method comprising:
 receiving, by one or more processors, a request to store a computer readable information object in network-accessible storage;   receiving, by one or more processors, a first data range associated with the computer readable information object;   determining, by one or more processors, that a size of the computer readable information object exceeds a predetermined size; and   evaluating, by one or more processors, whether to segment the computer readable information object, based at least on the determination that the size of the computer readable information object exceeds the predetermined size;   
       wherein, based on the evaluating, if the computer readable information object is to be segmented, further performing using one or more processors:
 segmenting the computer readable information object into a plurality of object segments, each segment corresponding to a portion of the computer readable information object; 
 generating a filename for each of the plurality of object segments, wherein each respective filename identifies the computer readable information object; 
 storing each of the plurality of object segments individually in network-accessible storage; 
 generating a manifest file related to the plurality of object segments; and 
 storing the manifest file in computer readable storage; 
 
       wherein the manifest file allows the plurality of object segments to be retrieved from network-accessible storage and to be reconstituted into a data range equal to the first data range; and 
       wherein if, based on the evaluating, the computer readable information object is not to be segmented, storing, by one or more processors, the first data range associated with the computer readable information object in network-accessible storage. 
     
     
         8 . The method of  claim 7 , further comprising:
 receiving, by one or more processors, a request to modify an object segment in the plurality of object segments, wherein the request includes a filename for the object segment in the plurality of object segments;   receiving, by one or more processors, the modification to the object segment in the plurality of object segments; and   updating, by one or more processors, the manifest file related to the plurality of object segments, based on the modification.   
     
     
         9 . The method of  claim 7 , wherein evaluating whether to segment the computer readable information object includes the step of examining the request for an indicia that the object should be segmented. 
     
     
         10 . The method of  claim 7 , wherein evaluating whether to segment the computer readable information object includes determining a file type of the computer readable information object, and wherein segmenting the computer readable information object into a plurality of object segments is performed based on the determined file type of the computer readable information object. 
     
     
         11 . The method of  claim 7 , wherein the filename further includes an identification of a respective object segment relative to other object segments segmented from the computer readable information object. 
     
     
         12 . The method of  claim 7 , wherein the manifest file further includes one or more filenames for each of the plurality of object segments. 
     
     
         13 . The method of  claim 7 , wherein segmenting the computer readable information object is performed before the data corresponding to the object is received. 
     
     
         14 . The method of  claim 7 , wherein the step of segmenting the computer readable information object occurs as the data corresponding to the computer readable information object is received. 
     
     
         15 . The method of  claim 7 , wherein the generated manifest file is individually addressable and manipulable. 
     
     
         16 . A non-transitory computer-readable storage medium storing program instructions that when executed by a processor cause the processor to implement a method for storing object data in a cloud storage system, the method comprising:
 receiving a request to store a computer readable information object in network-accessible storage;   receiving a first data range associated with the computer readable information object;   determining that a size of the computer readable information object exceeds a predetermined size; and   evaluating whether to segment the computer readable information object;   
       wherein, based on the evaluating, if the computer readable information object is to be segmented, further performing:
 segmenting the computer readable information object into a plurality of object segments, each segment corresponding to a portion of the computer readable information object; 
 generating a filename for each of the plurality of object segments, wherein each respective filename identifies the computer readable information object; 
 storing each of the plurality of object segments individually in network-accessible storage; 
 generating a manifest file related to the plurality of object segments; and 
 storing the manifest file in computer readable storage; 
 
       wherein the manifest file allows the plurality of object segments to be retrieved from network-accessible storage and to be reconstituted into a data range equal to the first data range; and 
       wherein if, based on the evaluating, the computer readable information object is not to be segmented, storing the first data range associated with the computer readable information object in network-accessible storage. 
     
     
         17 . The computer-readable storage medium of  claim 16 , the method further comprising:
 receiving a request to modify an object segment in the plurality of object segments, wherein the request includes a filename for the object segment in the plurality of object segments;   receiving the modification to the object segment in the plurality of object segments; and   updating the manifest file related to the plurality of object segments, based on the modification.   
     
     
         18 . The computer-readable storage medium of  claim 16 , wherein evaluating whether to segment the computer readable information object includes the step of examining the request for an indicia that the object should be segmented. 
     
     
         19 . The computer-readable storage medium of  claim 16 , wherein the filename further includes an identification of a respective object segment relative to other object segments segmented from the computer readable information object. 
     
     
         20 . The computer-readable storage medium of  claim 16 , wherein the manifest file further includes one or more filenames for each of the plurality of object segments. 
     
     
         21 . The computer-readable storage medium of  claim 16 , wherein the generated manifest file is individually addressable and manipulable.