Distributed Object Storage System Comprising Performance Optimizations

Abstract

A distributed object storage system comprises an encoding module configured to calculate for a plurality of predetermined values of the spreading requirement the cumulative size of the sub fragment files when stored on the file system with the predetermined block size; and select as a spreading requirement from said plurality of predetermined values a calculated value that is equal to one of said predetermined values for which the cumulative size is minimal.

What is claimed is: 
     
         1 . A distributed object storage system comprising:
 a plurality of storage nodes, each having a file system with a predetermined block size, the plurality of storage nodes comprising:
 a plurality of storage elements, configured to store and retrieve a data object comprising a predetermined number of sub fragments corresponding to a predetermined multiple of a desired spreading width; 
 wherein each one of the predetermined number of sub fragments comprises:
 encoded data of equal size, wherein the equal size of the encoded data is determined by dividing the data object by a factor equal to the predetermined multiple of the desired spreading width; and 
 decoding data, such that the data object can be decoded from a first number of the predetermined number of sub fragments, wherein the first number equals the predetermined multiple of the desired spreading width; and 
 
   at least one controller node, coupled with the plurality of storage nodes, the at least one controller node comprising:
 an encoding module configured to disassemble the data object into the predetermined number of sub fragments; and 
 a spreading module configured to store one or more sub fragment files on a number of the plurality of storage elements, wherein the number of the plurality of storage elements is equal to the desired spreading width; 
   wherein the encoding module is further configured to:
 calculate a plurality of real storage costs corresponding to a plurality of predetermined values of a spreading requirement and the predetermined block size, the plurality of real storage costs corresponding to a cumulative size of the sub fragment files, when encoded and stored on the file system; 
 responsive to the size of the data object being smaller than a predetermined lower data object size threshold, select the spreading requirement based on a corresponding real storage cost, calculated using a predetermined value of the plurality of predetermined values, for which the corresponding real storage cost is minimal; and 
 responsive to the size of the data object being equal to or larger than the predetermined lower data object size threshold, select as the spreading requirement a maximum value of the plurality of predetermined values of the spreading requirement. 
   
     
     
         2 . The distributed object storage system according to  claim 1 , wherein the spreading module is configured to group all sub fragments to be stored on a single storage element into a single sub fragment file. 
     
     
         3 . The distributed object storage system according to  claim 1 , wherein the plurality of predetermined values is an incremental range of integer values ranging from a predetermined minimum value to a predetermined maximum value. 
     
     
         4 . The distributed object storage system according to  claim 3 , wherein the predetermined minimum value is one. 
     
     
         5 . The distributed object storage system according to  claim 1 , wherein the lower data object size threshold is determined as a function of the block size of the file system in use on the storage elements. 
     
     
         6 . The distributed object storage system according to  claim 1 , further comprising a metadata storage comprising metadata of the data objects stored on the storage elements, the metadata comprising:
 a data object identifier; and   a list of identifiers of the storage elements on which sub fragments of the data object are stored.   
     
     
         7 . The distributed object storage system according to  claim 6 , wherein the metadata further comprises:
 an identifier for the calculated value that was selected as spreading requirement by the encoding module.   
     
     
         8 . The distributed object storage system according to  claim 6 , wherein the metadata further comprises an identifier of a type of storage and retrieval option that was used to store the data object. 
     
     
         9 . The distributed object storage system according to  claim 6 , wherein the metadata storage comprises at least one metadata parameter for a plurality of the data objects. 
     
     
         10 . The distributed object storage system according to  claim 8 , wherein the metadata comprises as a metadata parameter an identifier for a maximal concurrent failures tolerance. 
     
     
         11 . A method comprising:
 disassembling, by a controller node of a distributed object storage system, a data object into a predetermined number of sub fragments, the predetermined number of sub fragments comprising a data object identifier corresponding to the data object;
 wherein each one of the predetermined number of sub fragments comprises:
 encoded data of equal size, wherein the equal size of the encoded data is determined by dividing the data object by a factor equal to the predetermined multiple of the desired spreading width; and 
 decoding data, such that the data object can be decoded from a first number of the predetermined number of sub fragments, wherein the first number equals the predetermined multiple of the desired spreading width; 
 
   calculating, by the controller node of the distributed object storage system, a plurality of real storage costs, the plurality of real storage costs corresponding to a cumulative size of one or more sub fragment files, when encoded and stored on the distributed object storage system, corresponding to a plurality of predetermined values of a spreading requirement, wherein each sub fragment file comprises a portion of the predetermined number of sub fragments for storing on a single storage element;   responsive to the size of the data object being smaller than a predetermined lower data object size threshold, selecting, by the controller node of the distributed object storage system, the spreading requirement based on a corresponding real storage cost, calculated using a predetermined value of the plurality of predetermined values, for which the corresponding real storage cost is minimal;   responsive to the size of the data object being equal to or larger than the predetermined lower data object size threshold, selecting, by the controller node of the distributed object storage system, as the spreading requirement a maximum value of the plurality of predetermined values of the spreading requirement; and   storing, by the controller node of the distributed object storage system, the one or more sub fragment files on a number of a plurality of storage elements, wherein the number of the plurality of storage elements is equal to the desired spreading width.   
     
     
         12 . The method according to  claim 11 , wherein the plurality of predetermined values is an incremental range of integer values ranging from a predetermined minimum value to a predetermined maximum value. 
     
     
         13 . The method according to  claim 12 , wherein the predetermined minimum value is one. 
     
     
         14 . The method according to  claim 11 , wherein the lower data object size threshold is determined as a function of the block size of a file system in use on the storage elements. 
     
     
         15 . The method according to  claim 11 , further comprising:
 storing, by the controller node of the distributed object storage system, metadata of the data objects stored on the storage elements, the metadata comprising:
 the data object identifier; and 
 a list of identifiers of storage elements on which sub fragments of the data object are stored. 
   
     
     
         16 . A controller node of a distributed object storage system, the controller node comprising:
 means for disassembling a data object into a predetermined number of sub fragments, the predetermined number of sub fragments comprising a data object identifier corresponding to the data object
 wherein each one of the predetermined number of sub fragments comprises:
 encoded data of equal size, wherein the equal size of the encoded data is determined by dividing the data object by a factor equal to the predetermined multiple of the desired spreading width; and 
 decoding data, such that the data object can be decoded from a first number of the predetermined number of sub fragments, wherein the first number equals the predetermined multiple of the desired spreading width; 
 
   means for calculating a plurality of real storage costs, the plurality of real storage costs corresponding to a cumulative size of one or more sub fragment files, when encoded and stored on the distributed object storage system, wherein each sub fragment file comprises a portion of the predetermined number of sub fragments for storing on a single storage element;   means for selecting, responsive to the size of the data object being smaller than a predetermined lower data object size threshold, the spreading requirement based on a corresponding real storage cost, calculated using a predetermined value of the plurality of predetermined values, for which the corresponding real storage cost is minimal;   means for selecting, responsive to the size of the data object being equal to or larger than the predetermined lower data object size threshold, as the spreading requirement a maximum value of the plurality of predetermined values of the spreading requirement; and   means for storing the one or more sub fragment files on a number of a plurality of storage elements, wherein the number of the plurality of storage elements is equal to the desired spreading width.   
     
     
         17 . The controller node according to  claim 16 , wherein the plurality of predetermined values is an incremental range of integer values ranging from a predetermined minimum value to a predetermined maximum value. 
     
     
         18 . The controller node according to  claim 17 , wherein the predetermined minimum value is one. 
     
     
         19 . The controller node according to  claim 16 , wherein the lower data object size threshold is determined as a function of the block size of a file system in use on the storage elements. 
     
     
         20 . The controller node according to  claim 16 , further comprising means for storing metadata, the metadata comprising:
 a data object identifier; and   a list of identifiers of the storage elements on which sub fragments of the data object are stored.