Snapshot comparison with metadata compaction

Abstract

Snapshot or point-in-time image functionality improves the use of object-based datastores. An example system includes an object-based datastore and a metadata datastore associated with the object-based datastore. Instances of the metadata datastore are created as snapshot images of the object-based datastore. Comparison of snapshot images is important for database analytics, disaster recovery, data protection, and more. Example techniques provide comparison of snapshot images (as metadata datastore instances) and remain robust and accurate in view of compactions performed by the metadata datastore. An example technique includes generating and updating a graph-based data structure that captures relationships between metadata files in the metadata datastore, particularly between pre-compaction files and post-compaction files. The example technique further includes referencing the graph-based data structure to accelerate snapshot image comparison based on determining whether files of a source snapshot image were compacted into files of a destination snapshot image, and/or vice versa.

What is claimed is: 
     
         1 . A computer-implemented method for preventing compaction-based errors related to log-structure-merge-tree-based (LSM-based) snapshot images of a distributed object-based datastore, the computer-implemented method comprising:
 detecting, by one or more computing nodes associated with the distributed object-based datastore, compaction events performed by an LSM-based metadata datastore that stores metadata files for data objects included in the distributed object-based datastore,
 wherein each compaction event results in one or more first metadata files stored in the LSM-based metadata datastore being compacted into one or more new metadata files added to the LSM-based metadata datastore, 
 wherein each LSM-based snapshot image of the distributed object-based datastore references a respective subset of the metadata files stored in the LSM-based metadata datastore; 
   maintaining, by the one or more computing nodes, a directed acyclic graph (DAG) data structure that indicates, for each compaction event, a directed relationship from the one or more first metadata files to the one or more new metadata files;   receiving, by the one or more computing nodes, a request to compare a first LSM-based snapshot image of the distributed object-based datastore and a second LSM-based snapshot image of the distributed object-based datastore;   using, by the one or more computing nodes, the DAG data structure to filter the first LSM-based snapshot image and the second LSM-based snapshot image for certain metadata files referenced by the second LSM-based snapshot image that lack a respective directed relationship from any metadata file referenced by the first LSM-based snapshot image;   generating, by the one or more computing nodes, comparison data based on the certain metadata files; and   providing, by the one or more computing nodes, the comparison data in response to the request.   
     
     
         2 . The computer-implemented method of  claim 1 , further comprising:
 storing, by the one or more computing nodes, the comparison data in a snapshot-comparison database, the comparison data being associated with the first LSM-based snapshot image and the second LSM-based snapshot image.   
     
     
         3 . The computer-implemented method of  claim 2 , further comprising:
 receiving, by the one or more computing nodes, a second request that identifies the first LSM-based snapshot image and the second LSM-based snapshot image;   in response to the second request, retrieving, by the one or more computing nodes, the comparison data associated with the first LSM-based snapshot image and the second LSM-based snapshot image from the snapshot-comparison database; and   providing, by the one or more computing nodes, the comparison data in response to the second request.   
     
     
         4 . The computer-implemented method of  claim 2 , further comprising:
 receiving, by the one or more computing nodes, a second request that identifies a third LSM-based snapshot image and one of the first LSM-based snapshot image or the second LSM-based snapshot image;   in response to the second request, retrieving, by the one or more computing nodes, the comparison data associated with the first LSM-based snapshot image and the second LSM-based snapshot image;   using, by the one or more computing nodes, the comparison data to generate second comparison data for the third LSM-based snapshot image and the one of the first LSM-based snapshot image or the second LSM-based snapshot image; and   providing, by the one or more computing nodes, the comparison data in response to the second request.   
     
     
         5 . The computer-implemented method of  claim 1 , wherein the compaction events are detected via a synchronous event listener implemented by the one or more computing nodes, the synchronous event listener configured to receive, for a given compaction event, an indication of the one or more first metadata files and the one or more new metadata files from the LSM-based metadata datastore. 
     
     
         6 . The computer-implemented method of  claim 1 , further comprising:
 prior to generating the comparison data, estimating, by the one or more computing nodes, a size of the comparison data based on the first LSM-based snapshot image and the second LSM-based snapshot image; and   returning, by the one or more computing nodes, an error based on the size being greater than a threshold size.   
     
     
         7 . The computer-implemented method of  claim 1 , wherein the DAG data structure is configured to include a reference to each metadata file for each compaction event such that deletion of a given metadata file referenced by the DAG data structure is prevented. 
     
     
         8 . A distributed computing system comprising:
 a plurality of data nodes implementing an object-based datastore; and   one or more manager nodes each implementing a metadata database for the object-based datastore, wherein each manager node is configured to perform operations comprising:
 detecting compactions performed by a metadata database storing metadata files related to data objects of an object-based datastore, wherein each compaction results in an addition to the metadata database of one or more new metadata files that aggregate metadata information recorded on one or more pre-compaction metadata files stored in the metadata database; 
 receiving a request to compare a second snapshot image against a first snapshot image of the object-based datastore, wherein each of the first snapshot image and the second snapshot image references a corresponding subset of metadata files of the metadata database; 
 generating comparison data for the first snapshot image and the second snapshot image based on identifying particular metadata files referenced by the second snapshot image, the particular metadata files compacting the metadata information recorded on corresponding pre-compaction metadata files referenced by the first snapshot image; and 
 returning the comparison data in response to the request. 
   
     
     
         9 . The distributed computing system of  claim 8 , wherein the comparison data is further generated based on:
 determining that the particular metadata files referenced by the second snapshot image further compact the metadata information recorded on corresponding pre-compaction metadata files referenced by the first snapshot image with a separate metadata file that is not referenced by the first snapshot image or the second snapshot image, and   generating the comparison data using the separate metadata file instead of the particular metadata files.   
     
     
         10 . The distributed computing system of  claim 8 , wherein the operations further comprise:
 storing the comparison data in a snapshot-comparison database, the comparison data being associated with the first snapshot image and the second snapshot image.   
     
     
         11 . The distributed computing system of  claim 10 , further comprising:
 receiving a second request that identifies the first snapshot image and the second snapshot image;   in response to the second request, retrieving the comparison data associated with the first snapshot image and the second snapshot image from the snapshot-comparison database; and   providing the comparison data in response to the second request.   
     
     
         12 . The distributed computing system of  claim 10 , further comprising:
 receiving a second request that identifies a third snapshot image and one of the first snapshot image or the second snapshot image;   in response to the second request, retrieving the comparison data associated with the first snapshot image and the second snapshot image;   using the comparison data to generate second comparison data for the third snapshot image and the one of the first snapshot image or the second snapshot image; and   providing the comparison data in response to the second request.   
     
     
         13 . The distributed computing system of  claim 8 , wherein the compactions are detected via a synchronous event listener communicatively coupled with the metadata database, the synchronous event listener configured to receive, for a given compaction, an identification of the one or more new metadata files and the one or more pre-compaction metadata files. 
     
     
         14 . The distributed computing system of  claim 8 , further comprising:
 prior to generating the comparison data, estimating a size of the comparison data based on at least one of a respective creation time of the first snapshot image and the second snapshot image or a number of the particular metadata files identified with the second snapshot image; and   determining a time to generate the comparison data based on a prediction of resource availability according to the estimated size.   
     
     
         15 . The distributed computing system of  claim 8 , wherein the operations further comprise:
 storing a graph-based data structure in an active instance of the metadata database, the graph-based data structure recording directed relationships between the one or more pre-compaction metadata files and the one or more new metadata files of each compaction.   
     
     
         16 . At least one non-transitory computer-readable medium with instructions stored thereon that, when executed by a processor of a computing device, cause the computing device to perform operations comprising:
 detecting compactions performed by a metadata database storing metadata files related to data objects of an object-based datastore, wherein each compaction results in an addition to the metadata database of one or more new metadata files that aggregate metadata information recorded on a plurality of pre-compaction metadata files stored in the metadata database;   receiving a request to compare a second snapshot image of the object-based datastore against a first snapshot image of the object-based datastore, wherein each of the first snapshot image and the second snapshot image references a respective subset of the metadata files in the metadata database;   generating comparison data for the first snapshot image and the second snapshot image using certain metadata files referenced by the second snapshot image that are not identified as new metadata files that correspond to pre-compaction metadata files referenced by the first snapshot image; and   returning the comparison data in response to the request.   
     
     
         17 . The at least one non-transitory computer-readable medium of  claim 16 , wherein the operations further comprise:
 recording a given compaction based on (i) including, in a graph-based data structure, a hard-link reference to each of the one or more new metadata files and the plurality of pre-compaction metadata files, and (ii) associating, via the graph-based data structure, the one or more new metadata files with the plurality of pre-compaction metadata files, wherein the metadata database is configured to remove a given metadata file that has zero hard-link references.   
     
     
         18 . The at least one non-transitory computer-readable medium of  claim 16 , wherein the operations further comprise:
 prior to generating the comparison data, estimating a size of the comparison data based on a number of the certain metadata files; and   determining whether to handle the request based on the estimated size.   
     
     
         19 . The at least one non-transitory computer-readable medium of  claim 16 , wherein the operations further comprise:
 storing the comparison data in a snapshot-comparison database in association with the first snapshot image and the second snapshot image; and   in response to a second request that specifies the first snapshot image and the second snapshot image, retrieving and returning the comparison data.   
     
     
         20 . The at least one non-transitory computer-readable medium of  claim 16 , wherein the operations further comprise:
 determining to generate the comparison data for the first snapshot image and the second snapshot image based on a distance between the first snapshot image and the second snapshot image on a snapshot creation order satisfying a threshold distance.