Decentralized edge storage network with flexible file sharding

Abstract

Methods and systems for decentralized storage are provided. To store a file in a decentralized network, a source node in the decentralized network first divides the file into a plurality of file portions. For each given file portion, a corresponding peer storage node is located in the decentralized network, where a distance between the corresponding peer storage node and the given file portion is below a given threshold, where the distance is computed based on an identifier of the corresponding peer storage node, and an identifier of the given file portion, and where the given threshold is associated with a storage capacity of the corresponding peer storage node. The source node then transmits each given file portion to the corresponding peer storage node, and generates a file identifier for the file, based on the identifiers of the plurality of file portions.

What is claimed is: 
     
       1. A method for decentralized storage, comprising:
 dividing, by a source node in a decentralized network, a file into a plurality of file portions; 
 for each given file portion in the plurality of file portions, locating a corresponding peer storage node in the decentralized network,
 wherein a distance between the corresponding peer storage node and the given file portion is below a given threshold, 
 wherein the distance is computed based on an identifier of the corresponding peer storage node and an identifier of the given file portion, and 
 wherein the given threshold is associated with a storage capacity of the corresponding peer storage node; 
 
 transmitting, by the source node, each given file portion to the corresponding peer storage node; and 
 generating, by the source node, a file identifier for the file, based on the identifiers of the plurality of file portions. 
 
     
     
       2. The method of  claim 1 , wherein for each given file portion in the plurality of file portions, the locating the corresponding peer storage node in the decentralized network comprises:
 gossiping the identifier of the given file portion through the decentralized network; and 
 receiving a location notification from the corresponding peer storage node, wherein the location notification indicates that the distance between the corresponding peer storage node and the given file portion is below the given threshold associated with the storage capacity of the corresponding peer storage node. 
 
     
     
       3. The method of  claim 1 , wherein for each given file portion in the plurality of file portions, the locating the corresponding peer storage node in the decentralized network comprises:
 retrieving, by the source node, a list of registered peer storage nodes from a blockchain; and 
 selecting the corresponding peer storage node from the list of registered peer storage nodes. 
 
     
     
       4. The method of  claim 1 , wherein the identifier of the corresponding peer storage node and the identifier of the given file portion are computed using a hash function. 
     
     
       5. The method of  claim 1 , wherein the identifier of the corresponding peer storage node is computed based on a crypto wallet address associated with the corresponding peer storage node. 
     
     
       6. The method of  claim 1 , wherein the generating of the file identifier comprises performing a vector commitment scheme on the identifiers of the plurality of file portions. 
     
     
       7. The method of  claim 1 , further comprising:
 submitting the file identifier to a smart contract. 
 
     
     
       8. The method of  claim 1 , further comprising:
 performing a distributed hash table (DHT) lookup using the file identifier, to determine the identifier of each of the plurality of file portions; 
 for each given file portion of the plurality of file portions,
 performing a DHT lookup on the identifier of the given file portion to determine an address for a peer storage node storing the given file portion; 
 retrieving each given file portion from the peer storage node storing the given file portion, using the determined address; and 
 
 assembling the retrieved file portions into the file. 
 
     
     
       9. A source node system in a decentralized network for decentralized storage, comprising:
 at least one processor; and 
 a non-transitory physical medium for storing program code accessible by the processor, the program code when executed by the processor causes the processor to:
 divide a file into a plurality of file portions; 
 for each given file portion in the plurality of file portions, locate a corresponding peer storage node in the decentralized network,
 wherein a distance between the corresponding peer storage node and the given file portion is below a given threshold, 
 wherein the distance is computed based on an identifier of the corresponding peer storage node and an identifier of the given file portion, and 
 wherein the given threshold is associated with a storage capacity of the corresponding peer storage node; 
 
 transmit each given file portion to the corresponding peer storage node; and 
 generate a file identifier for the file, based on the identifiers of the plurality of file portions. 
 
 
     
     
       10. The source node system of  claim 9 , wherein for each given file portion in the plurality of file portions, the program code to locate the corresponding peer storage node, when executed by the processor, causes the processor to:
 gossip the identifier of the given file portion through the decentralized network; and 
 receive a location notification from the corresponding peer storage node, wherein the location notification indicates that the distance between the corresponding peer storage node and the given file portion is below the given threshold associated with the storage capacity of the corresponding peer storage node. 
 
     
     
       11. The source node system of  claim 9 , wherein for each given file portion in the plurality of file portions, the program code to locate the corresponding peer storage node, when executed by the processor, causes the processor to:
 retrieve a list of registered peer storage nodes from a blockchain; and 
 select the corresponding peer storage node from the list of registered peer storage nodes. 
 
     
     
       12. The source node system of  claim 9 , wherein the identifier of the corresponding peer storage node and the identifier of the given file portion is computed using a hash function. 
     
     
       13. The source node system of  claim 9 , wherein the identifier of the corresponding peer storage node is computed based on a crypto wallet address associated with the corresponding peer storage node. 
     
     
       14. The source node system of  claim 9 , wherein the program code to generate the file identifier, when executed by the processor, causes the processor to:
 perform a vector commitment scheme on the identifiers of the plurality of file portions. 
 
     
     
       15. The source node system of  claim 9 , wherein the program code when executed by the processor further causes the processor to:
 submit the file identifier to a smart contract. 
 
     
     
       16. The source node system of  claim 9 , wherein the program code when executed by the processor further causes the processor to:
 perform a distributed hash table (DHT) lookup using the file identifier, to determine the identifier of each of the plurality of file portions; 
 for each given file portion of the plurality of file portions,
 perform a DHT lookup on the identifier of the given file portion to determine an address for a peer storage node storing the given file portion; 
 retrieve each given file portion from the peer storage node storing the given file portion, using the determined address; and 
 
 assemble the retrieved file portions into the file. 
 
     
     
       17. A method for decentralized storage, comprising:
 receiving, by a peer storage node, a file portion of a file, wherein the file is divided into a plurality of file portions by a source node; 
 computing, by the peer storage node, a distance based on an identifier of the received file portion, and an identifier of the peer storage node; 
 determining, by the peer storage node, whether the distance is below a threshold associated with a storage capacity of the peer storage node; and 
 storing the received file portion in the peer storage node in response to determining that the distance is below the threshold. 
 
     
     
       18. The method of  claim 17 , wherein the identifier of the peer storage node and the identifier of the received file portion are computed using a hash function. 
     
     
       19. The method of  claim 17 , wherein the identifier of the peer storage node is computed based on a crypto wallet address associated with the peer storage node. 
     
     
       20. The method of  claim 17 , further comprising:
 retrieving a random number from a blockchain; 
 computing a reward hash, based on the received file portion and the random number; and 
 submitting the reward hash to a smart contract on the blockchain.