In-place erasure code transcoding for distributed file systems

Abstract

Embodiments are directed to in-place erasure code transcoding for distributed file systems. A file system may be divided into a first partition associated with a first erasure code and a second partition second partition is associated with a second erasure code. If the second partition has sufficient storage space to store protection groups further actions may be performed, including: determining block stores in the first partition associated with the protection groups; transcoding contents of the block stores into other block stores based on the second erasure code; storing the other block stores in the second partition; deleting the block stores from the first partition; shifting another portion of the storage space from the first partition to the second partition such that the shifted other portion increases a size of the second partition to provide sufficient storage space for other protection groups; or the like.

What is claimed is: 
     
       1. A method for managing data over a network using one or more processors that execute instructions to perform actions, comprising:
 providing a file system that includes a plurality of protection groups that each are associated with a plurality of block stores, wherein each block store associated with the plurality of protection groups is encoded based on a first erasure code and a first block store size; 
 storing each of the one or more block stores in one or more of plurality of slots that are defined for separately storing a block store on a different one of a plurality of storage devices that are separately located in the file system; 
 dividing the file system into a first partition and a second partition, wherein a portion of a storage space in the file system is shifted from the first partition to the second partition to provide sufficient storage space for one or more of the plurality of protection groups in the second partition based on a specification for a second erasure code, and wherein the first partition is associated with the first erasure code and the second partition is associated with the second erasure code; 
 in response to adding one or more additional storage devices to the storage space of the file system, providing additional storage space to the second partition; 
 in response to the second partition having sufficient storage space to store one or more of the plurality of protection groups, performing further actions, including:
 determining one or more block stores in the first partition associated with the one or more protection groups; 
 transcoding contents of the one or more block stores into one or more other block stores based on the second erasure code and a second block store size, wherein the one or more other block stores are sized based on the second block store size for the second erasure code; 
 storing the one or more other block stores in the second partition, wherein the one or more other block stores are stored in one or more of plurality of slots that are defined for separately storing the one or more other block stores on different storage devices that are separately located in the file system; 
 deleting the one or more block stores from the first partition; and 
 shifting another portion of the storage space from the first partition to the second partition, wherein the shifted other portion increases a size of the second partition to provide sufficient storage space for one or more other protection groups and the first partition is decreased in size based on the deletion of the one or more block stores; and 
 
 in response to the second partition reaching a maximum size, reporting that the file system is encoded based on the second erasure code. 
 
     
     
       2. The method of  claim 1 , further comprising:
 providing one or more client requests to the file system; and 
 satisfying the one or more client requests using contents from one or more of the first partition or the second partition. 
 
     
     
       3. The method of  claim 1 , further comprising:
 moving a portion of plurality of the block stores in the first partition to provide storage space for the second partition. 
 
     
     
       4. The method of  claim 1 , further comprising:
 in response to one or more storage devices being added to the file system, distributing a portion of the plurality of block stores to the one or more added storage devices to provide working space in the file system to transcode the plurality of block stores based on the second erasure code. 
 
     
     
       5. The method of  claim 1 , wherein transcoding the contents of the one or more block stores into one or more other block stores, further comprises:
 disassociating the one or more block stores from the one or more protection groups; and 
 associating the one or more other block stores with the one or more protection groups, wherein a storage capacity, protection level, and availability of each of the one or more protection groups remains unchanged. 
 
     
     
       6. The method of  claim 1 , further comprising:
 in response to modifying the file system with one or more additional storage devices, performing additional actions, including:
 determining a storage inefficiency value for the modified file system based on the first erasure code; 
 determining a data loss risk value for the modified file system based on the first erasure code; and 
 in response to one or more of the storage inefficiency value or the data loss risk value exceeding a threshold value, converting the modified file system to protect its contents based on the second erasure code. 
 
 
     
     
       7. A system for managing data comprising:
 a network computer, comprising:
 a memory that stores at least instructions; and 
 one or more processors that execute instructions that are configured to cause performance of actions, including:
 providing a file system that includes a plurality of protection groups that each are associated with a plurality of block stores, wherein each block store associated with the plurality of protection groups is encoded based on a first erasure code and a first block store size; 
 storing each of the one or more block stores in one or more of plurality of slots that are defined for separately storing a block store on a different one of a plurality of storage devices that are separately located in the file system; 
 dividing the file system into a first partition and a second partition, wherein a portion of a storage space in the file system is shifted from the first partition to the second partition to provide sufficient storage space for one or more of the plurality of protection groups in the second partition based on a specification for a second erasure code, and wherein the first partition is associated with the first erasure code and the second partition is associated with the second erasure code; 
 in response to adding one or more additional storage devices to the storage space of the file system, providing additional storage space to the second partition; 
 in response to the second partition having sufficient storage space to store one or more of the plurality of protection groups, performing further actions, including:
 determining one or more block stores in the first partition associated with the one or more protection groups; 
 transcoding contents of the one or more block stores into one or more other block stores based on the second erasure code and a second block store size, wherein the one or more other block stores are sized based on the second block store size for the second erasure code; 
 storing the one or more other block stores in the second partition, wherein the one or more other block stores are stored in one or more of plurality of slots that are defined for separately storing the one or more other block stores on different storage devices that are separately located in the file system; 
 deleting the one or more block stores from the first partition; and 
 shifting another portion of the storage space from the first partition to the second partition, wherein the shifted other portion increases a size of the second partition to provide sufficient storage space for one or more other protection groups and the first partition is decreased in size based on the deletion of the one or more block stores; and 
 
 in response to the second partition reaching a maximum size, reporting that the file system is encoded based on the second erasure code; and 
 
 
 a client computer, comprising:
 a memory that stores at least instructions; and 
 one or more processors that execute instructions that are configured to cause performance of actions, including, providing one or more of one or more client requests or information associated with the second erasure code. 
 
 
     
     
       8. The system of  claim 7 , wherein the one or more network computer processors execute instructions that perform actions, further comprising:
 providing one or more client requests to the file system; and 
 satisfying the one or more client requests using contents from one or more of the first partition or the second partition. 
 
     
     
       9. The system of  claim 7 , wherein the one or more network computer processors execute instructions that perform actions, further comprising:
 moving a portion of plurality of the block stores in the first partition to provide storage space for the second partition. 
 
     
     
       10. The system of  claim 7 , wherein the one or more network computer processors execute instructions that perform actions, further comprising:
 in response to one or more storage devices being added to the file system, distributing a portion of the plurality of block stores to the one or more added storage devices to provide working space in the file system to transcode the plurality of block stores based on the second erasure code. 
 
     
     
       11. The system of  claim 7 , wherein transcoding the contents of the one or more block stores into one or more other block stores, further comprises:
 disassociating the one or more block stores from the one or more protection groups; and 
 associating the one or more other block stores with the one or more protection groups, wherein a storage capacity, protection level, and availability of each of the one or more protection groups remains unchanged. 
 
     
     
       12. The system of  claim 7 , wherein the one or more network computer processors execute instructions that perform actions, further comprising:
 in response to modifying the file system with one or more additional storage devices, performing additional actions, including:
 determining a storage inefficiency value for the modified file system based on the first erasure code; 
 determining a data loss risk value for the modified file system based on the first erasure code; and 
 in response to one or more of the storage inefficiency value or the data loss risk value exceeding a threshold value, converting the modified file system to protect its contents based on the second erasure code. 
 
 
     
     
       13. A processor readable non-transitory storage media that includes instructions that are configured to cause actions for managing data in over a network, wherein execution of the instructions by one or more processors on one or more network computers performs actions, comprising:
 providing a file system that includes a plurality of protection groups that each are associated with a plurality of block stores, wherein each block store associated with the plurality of protection groups is encoded based on a first erasure code and a first block store size; 
 storing each of the one or more block stores in one or more of plurality of slots that are defined for separately storing a block store on a different one of a plurality of storage devices that are separately located in the file system; 
 dividing the file system into a first partition and a second partition, wherein a portion of a storage space in the file system is shifted from the first partition to the second partition to provide sufficient storage space for one or more of the plurality of protection groups in the second partition based on a specification for a second erasure code, and wherein the first partition is associated with the first erasure code and the second partition is associated with the second erasure code; 
 in response to adding one or more additional storage devices to the storage space of the file system, providing additional storage space to the second partition; 
 in response to the second partition having sufficient storage space to store one or more of the plurality of protection groups, performing further actions, including:
 determining one or more block stores in the first partition associated with the one or more protection groups; 
 transcoding contents of the one or more block stores into one or more other block stores based on the second erasure code and a second block store size, wherein the one or more other block stores are sized based on the second block store size for the second erasure code; 
 storing the one or more other block stores in the second partition, wherein the one or more other block stores are stored in one or more of plurality of slots that are defined for separately storing the one or more other block stores on different storage devices that are separately located in the file system; 
 deleting the one or more block stores from the first partition; and 
 shifting another portion of the storage space from the first partition to the second partition, wherein the shifted other portion increases a size of the second partition to provide sufficient storage space for one or more other protection groups and the first partition is decreased in size based on the deletion of the one or more block stores; and 
 
 in response to the second partition reaching a maximum size, reporting that the file system is encoded based on the second erasure code. 
 
     
     
       14. The media of  claim 13 , further comprising:
 providing one or more client requests to the file system; and 
 satisfying the one or more client requests using contents from one or more of the first partition or the second partition. 
 
     
     
       15. The media of  claim 13 , further comprising:
 moving a portion of plurality of the block stores in the first partition to provide storage space for the second partition. 
 
     
     
       16. The media of  claim 13 , further comprising:
 in response to one or more storage devices being added to the file system, distributing a portion of the plurality of block stores to the one or more added storage devices to provide working space in the file system to transcode the plurality of block stores based on the second erasure code. 
 
     
     
       17. The media of  claim 13 , wherein transcoding the contents of the one or more block stores into one or more other block stores, further comprises:
 disassociating the one or more block stores from the one or more protection groups; and 
 associating the one or more other block stores with the one or more protection groups, wherein a storage capacity, protection level, and availability of each of the one or more protection groups remains unchanged. 
 
     
     
       18. The media of  claim 13 , further comprising:
 in response to modifying the file system with one or more additional storage devices, performing additional actions, including:
 determining a storage inefficiency value for the modified file system based on the first erasure code; 
 determining a data loss risk value for the modified file system based on the first erasure code; and 
 in response to one or more of the storage inefficiency value or the data loss risk value exceeding a threshold value, converting the modified file system to protect its contents based on the second erasure code. 
 
 
     
     
       19. A network computer for managing data, comprising:
 a memory that stores at least instructions; and 
 one or more processors that execute instructions that are configured to cause performance of actions, including:
 providing a file system that includes a plurality of protection groups that each are associated with a plurality of block stores, wherein each block store associated with the plurality of protection groups is encoded based on a first erasure code and a first block store size; 
 storing each of the one or more block stores in one or more of plurality of slots that are defined for separately storing a block store on a different one of a plurality of storage devices that are separately located in the file system; 
 dividing the file system into a first partition and a second partition, wherein a portion of a storage space in the file system is shifted from the first partition to the second partition to provide sufficient storage space for one or more of the plurality of protection groups in the second partition based on a specification for a second erasure code, and wherein the first partition is associated with the first erasure code and the second partition is associated with the second erasure code; 
 in response to adding one or more additional storage devices to the storage space of the file system, providing additional storage space to the second partition; 
 in response to the second partition having sufficient storage space to store one or more of the plurality of protection groups, performing further actions, including:
 determining one or more block stores in the first partition associated with the one or more protection groups; 
 transcoding contents of the one or more block stores into one or more other block stores based on the second erasure code and a second block store size, wherein the one or more other block stores are sized based on the second block store size for the second erasure code; 
 storing the one or more other block stores in the second partition, wherein the one or more other block stores are stored in one or more of plurality of slots that are defined for separately storing the one or more other block stores on different storage devices that are separately located in the file system; 
 deleting the one or more block stores from the first partition; and 
 shifting another portion of the storage space from the first partition to the second partition, wherein the shifted other portion increases a size of the second partition to provide sufficient storage space for one or more other protection groups and the first partition is decreased in size based on the deletion of the one or more block stores; and 
 
 in response to the second partition reaching a maximum size, reporting that the file system is encoded based on the second erasure code. 
 
 
     
     
       20. The network computer of  claim 19 , wherein the one or more processors execute instructions that perform actions further comprising:
 providing one or more client requests to the file system; and 
 satisfying the one or more client requests using contents from one or more of the first partition or the second partition. 
 
     
     
       21. The network computer of  claim 19 , wherein the one or more processors execute instructions that perform actions further comprising:
 moving a portion of plurality of the block stores in the first partition to provide storage space for the second partition. 
 
     
     
       22. The network computer of  claim 19 , wherein the one or more processors execute instructions that perform actions further comprising:
 in response to one or more storage devices being added to the file system, distributing a portion of the plurality of block stores to the one or more added storage devices to provide working space in the file system to transcode the plurality of block stores based on the second erasure code. 
 
     
     
       23. The network computer of  claim 19 , wherein transcoding the contents of the one or more block stores into one or more other block stores, further comprises:
 disassociating the one or more block stores from the one or more protection groups; and 
 associating the one or more other block stores with the one or more protection groups, wherein a storage capacity, protection level, and availability of each of the one or more protection groups remains unchanged. 
 
     
     
       24. The network computer of  claim 19 , wherein the one or more processors execute instructions that perform actions further comprising:
 in response to modifying the file system with one or more additional storage devices, performing additional actions, including:
 determining a storage inefficiency value for the modified file system based on the first erasure code; 
 determining a data loss risk value for the modified file system based on the first erasure code; and 
 in response to one or more of the storage inefficiency value or the data loss risk value exceeding a threshold value, converting the modified file system to protect its contents based on the second erasure code.