Synchronizing replicas with media errors in distributed storage systems

Abstract

Systems and techniques are described for transferring data. A described technique includes receiving a request to transmit a data block from a first data storage device to a second data storage device. An attempt to read the data block from the first data storage device is made. A media error resulting from the attempt to read the data block from the first data storage device is detected. In response to detecting the media error, a new data block is generated and includes mismatched checksum data that causes a checksum mismatched error when the new data block is accessed. The new data block is transmitted for storage at the second data storage device in place of the data block.

What is claimed is: 
     
       1. A method comprising:
 receiving a request to copy a replica of an object from a first host to a second host, wherein the first host stores data of the replica of the object in a set of data blocks; 
 for each of a plurality of subsets of the set of data blocks, determining whether the subset of data blocks has a media error by attempting to read the subset of data blocks; 
 for each subset of data blocks that does not have a media error, copying the subset of data blocks from the first host to the second host; 
 for each subset of data blocks that has a media error:
 identifying, in the subset of data blocks, each data block that has a media error; 
 generating, for each identified data block, a new data block including mismatched checksum data that causes a checksum mismatched error when the new data block is accessed; and 
 copying, to the second host, (i) each data block of the subset of data blocks that does not have a media error and (ii) the new data block generated for each identified data block. 
 
 
     
     
       2. The method of  claim 1 , wherein identifying, in the subset of data blocks, each data block that has a media error comprises attempting to read each data block in the subset of data blocks. 
     
     
       3. The method of  claim 1 , wherein determining whether the subset of data blocks has a media error by attempting to read the subset of data blocks comprises:
 attempting to read at least one of payload data or checksum data of the subset of data blocks; 
 receiving an error message in response to attempting to read the at least one of payload data or checksum data of the subset of data blocks; and 
 in response to receiving the error message, determining that the subset of data blocks has a media error. 
 
     
     
       4. The method of  claim 1 , wherein each new data block includes all-zero payload data. 
     
     
       5. The method of  claim 1 , further comprising:
 receiving a request to access payload data from a particular new data block stored at the second host; 
 generating checksum data for the particular new data block by applying a checksum function to the payload data of the particular new data block stored at the second host; 
 determining that mismatched checksum data of the particular new data block stored at the second host does not match the generated checksum data; and 
 generating a checksum mismatched error in response to determining that mismatched checksum data of the particular new data block stored at the second host does not match the generated checksum data. 
 
     
     
       6. The method of  claim 1 , further comprising:
 receiving a request to write data to a particular new data block stored at the second host; 
 overwriting payload data of the particular new data block at the second host with the data of the received request; 
 receiving, subsequent to the request to write data to the particular new data block, a request to read data from the particular new data block at the second host; and 
 providing the payload data of the particular new data block in response to the request to read data from the particular new data block. 
 
     
     
       7. The method of  claim 1 , wherein copying the new data block in place of each data block that has a media error block prevents interruption of copying the replica of the object from the first host to the second host. 
     
     
       8. A system comprising:
 a data processing apparatus; and 
 a memory apparatus in data communication with the data processing apparatus and storing instructions executable by the data processing apparatus and that upon such execution cause the data processing apparatus to perform operations comprising:
 receiving a request to copy a replica of an object from a first host to a second host, wherein the first host stores data of the replica of the object in a set of data blocks; 
 for each of a plurality of subsets of the set of data blocks, determining whether the subset of data blocks has a media error by attempting to read the subset of data blocks; 
 for each subset of data blocks that does not have a media error, copying the subset of data blocks from the first host to the second host; 
 for each subset of data blocks that has a media error:
 identifying, in the subset of data blocks, each data block that has a media error; 
 generating, for each identified data block, a new data block including mismatched checksum data that causes a checksum mismatched error when the new data block is accessed; and 
 copying, to the second host, (i) each data block of the subset of data blocks that does not have a media error and (ii) the new data block generated for each identified data block. 
 
 
 
     
     
       9. The system of  claim 8 , wherein identifying, in the subset of data blocks, each data block that has a media error comprises attempting to read each data block in the subset of data blocks. 
     
     
       10. The system of  claim 8 , wherein determining whether the subset of data blocks has a media error by attempting to read the subset of data blocks comprises:
 attempting to read at least one of payload data or checksum data of the subset of data blocks; 
 receiving an error message in response to attempting to read the at least one of payload data or checksum data of the subset of data blocks; and 
 in response to receiving the error message, determining that the subset of data blocks has a media error. 
 
     
     
       11. The system of  claim 8 , wherein each new data block includes all-zero payload data. 
     
     
       12. The system of  claim 8 , wherein the operations comprise:
 receiving a request to access payload data from a particular new data block stored at the second host; 
 generating checksum data for the particular new data block by applying a checksum function to the payload data of the particular new data block stored at the second host; 
 determining that mismatched checksum data of the particular new data block stored at the second host does not match the generated checksum data; and 
 generating a checksum mismatched error in response to determining that mismatched checksum data of the particular new data block stored at the second host does not match the generated checksum data. 
 
     
     
       13. The system of  claim 8 , wherein the operations comprise:
 receiving a request to write data to a particular new data block stored at the second host; 
 overwriting payload data of the particular new data block at the second host with the data of the received request; 
 receiving, subsequent to the request to write data to the particular new data block, a request to read data from the particular new data block at the second host; and 
 providing the payload data of the particular new data block in response to the request to read data from the particular new data block. 
 
     
     
       14. The system of  claim 8 , wherein copying the new data block in place of each data block that has a media error block prevents interruption of copying the replica of the object from the first host to the second host. 
     
     
       15. A non-transitory computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:
 receiving a request to copy a replica of an object from a first host to a second host, wherein the first host stores data of the replica of the object in a set of data blocks; 
 for each of a plurality of subsets of the set of data blocks, determining whether the subset of data blocks has a media error by attempting to read the subset of data blocks; 
 for each subset of data blocks that does not have a media error, copying the subset of data blocks from the first host to the second host; 
 for each subset of data blocks that has a media error:
 identifying, in the subset of data blocks, each data block that has a media error; 
 generating, for each identified data block, a new data block including mismatched checksum data that causes a checksum mismatched error when the new data block is accessed; and 
 copying, to the second host, (i) each data block of the subset of data blocks that does not have a media error and (ii) the new data block generated for each identified data block. 
 
 
     
     
       16. The non-transitory computer storage medium of  claim 15 , wherein identifying, in the subset of data blocks, each data block that has a media error comprises attempting to read each data block in the subset of data blocks. 
     
     
       17. The non-transitory computer storage medium of  claim 15 , wherein determining whether the subset of data blocks has a media error by attempting to read the subset of data blocks comprises:
 attempting to read at least one of payload data or checksum data of the subset of data blocks; 
 receiving an error message in response to attempting to read the at least one of payload data or checksum data of the subset of data blocks; and 
 in response to receiving the error message, determining that the subset of data blocks has a media error. 
 
     
     
       18. The non-transitory computer storage medium of  claim 15 , wherein each new data block includes all-zero payload data. 
     
     
       19. The non-transitory computer storage medium of  claim 15 , wherein the operations comprise:
 receiving a request to access payload data from a particular new data block stored at the second host; 
 generating checksum data for the particular new data block by applying a checksum function to the payload data of the particular new data block stored at the second host; 
 determining that mismatched checksum data of the particular new data block stored at the second host does not match the generated checksum data; and 
 generating a checksum mismatched error in response to determining that mismatched checksum data of the particular new data block stored at the second host does not match the generated checksum data. 
 
     
     
       20. The non-transitory computer storage medium of  claim 15 , wherein the operations comprise:
 receiving a request to write data to a particular new data block stored at the second host; 
 overwriting payload data of the particular new data block at the second host with the data of the received request; 
 receiving, subsequent to the request to write data to the particular new data block, a request to read data from the particular new data block at the second host; and 
 providing the payload data of the particular new data block in response to the request to read data from the particular new data block.