Metadata recovery for de-duplicated data

Abstract

A data stream is stored in storage media. As part of the storage, the data stream is divided into a plurality of chunks. The plurality of chunks include a target chunk that is next to a first chunk in a file within the data stream. A determination is made that the target chunk matches an existing chunk stored in the storage media. In response to the determination, a first pointer to the existing stored chunk is created in file metadata for the file. Also in response to the determination, a second pointer to a first stored chunk that matches the first chunk is created in chunk metadata embedded with the existing stored chunk.

1 . A method comprising:
 storing a data stream in storage media, wherein the storing the data stream comprises:
 dividing a file within the data stream into a plurality of chunks, the plurality of chunks including a first chunk that is an initial chunk in the file that is followed, in the file, by a second chunk that is followed, in the file, by a third chunk that is followed, in the file, by a fourth chunk that is followed, in the file, by a fifth chunk that is a final chunk in the file; 
 determining that the first chunk matches a first existing chunk stored in the storage media; 
 creating, in response to the determining that the first chunk matches the first existing stored chunk, a first pointer to the first existing stored chunk in file metadata for the file; 
 creating a first recovery bit and a null value in a first record in chunk metadata embedded with the first existing stored chunk; 
 determining that the second chunk does not match any existing chunk in the storage media; 
 storing, in response to the determining that the second chunk does not match any existing stored chunks, the second chunk in the storage media as a second stored chunk; 
 creating a second pointer to the second stored chunk in the file metadata for the file; 
 creating a second recovery bit and a third pointer to the first existing stored chunk in a second record in chunk metadata embedded with the second stored chunk; 
 determining that the third chunk matches a third existing chunk stored in the storage media; 
 creating, in response to the determining that the third chunk matches the third existing stored chunk, a fourth pointer to the third existing stored chunk in file metadata for the file; 
 creating a third recovery bit and a fifth pointer to the second stored chunk in a third record in chunk metadata embedded with the third existing stored chunk; 
 determining that the fourth chunk matches the second stored chunk; 
 creating, in response to the determining that the fourth chunk matches the second stored chunk, a sixth pointer to the second stored chunk in file metadata for the file; 
 creating a fourth recovery bit and a seventh pointer to the third existing stored chunk in a fourth record in chunk metadata embedded with the second stored chunk; 
 determining that the fifth chunk matches a fourth existing chunk stored in the storage media; 
 creating, in response to the determining that the fifth chunk matches the fourth existing stored chunk, an eighth pointer to the fourth existing stored chunk in file metadata for the file; and 
 creating a fifth recovery bit, a final chunk indicator value, and a ninth pointer to the second stored chunk in a fifth record in chunk metadata embedded with the fourth existing stored chunk; 
   detecting that the file metadata is lost or corrupted; and   performing, in response to the detecting, a recovery operation, wherein the recovery operation comprises:
 reading the fifth record in chunk metadata embedded with the fourth existing stored chunk; 
 determining, based on reading the final chunk indicator value and the ninth pointer in the fifth record in chunk metadata embedded with the fourth existing stored chunk, that the fifth chunk does not have any following chunks in the file and that the fifth chunk follows the fourth chunk in the file; 
 setting the fifth recovery bit; 
 reading the fourth record in chunk metadata embedded with the second stored chunk; 
 determining, based on reading the seventh pointer in the fourth record in chunk metadata embedded with the second stored chunk, that the fourth chunk follows the third chunk in the file; 
 setting the fourth recovery bit; 
 reading the third record in chunk metadata embedded with the third existing stored chunk; 
 determining, based on reading the fifth pointer in the third record in chunk metadata embedded with the third existing stored chunk, that the third chunk follows the second chunk in the file; 
 setting the third recovery bit; 
 reading the second record in chunk metadata embedded with the second stored chunk; 
 determining, based on reading the third pointer in the second record in chunk metadata embedded with the second stored chunk, that the second chunk follows the first chunk in the file; 
 setting the second recovery bit; 
 reading the first record in chunk metadata embedded with the first existing stored chunk; 
 determining, based on reading the null value in the first record in the chunk metadata embedded with the first existing stored chunk, that the first chunk does not have any preceding chunks in the file; 
 setting the first recovery bit; and 
 creating a recovered version of the file metadata, wherein, based on the determining that the target chunk does not have any preceding chunks in the file, the determining that the second chunk follows the first chunk in the file, the determining that the third chunk follows the second chunk in the file, the determining that the fourth chunk follows the third chunk in the file, the determining that the fifth chunk follows the fourth chunk in the file, and the determining that the fifth chunk does not have any following chunks in the file, the recovered version of the file metadata indicates that the first chunk is the initial chunk in the file that is followed, in the file, by the second chunk that is followed, in the file, by the third chunk that is followed, in the file, by the fourth chunk that is followed, in the file, by the fifth chunk that is the final chunk in the file.