Method and system for idempotent synthetic full backups in storage devices

Abstract

In general, embodiments relate to a method for generating synthetic full backups, the method comprising: performing a verification that a previous backup of source data stored in a data domain is a failed synthetic full backup, obtaining based on the verification a latest snapshot of the source data, obtaining based on the verification a prior snapshot of the source data making a determination, using a copy list that a first portion of the data items in the copy list exists in the previous backup and a second portion of the data items does not exist in the previous backup, and performing based on the determination a copy operation to copy the second portion of the data items to the data domain to obtain a synthetic full backup.

What is claimed is: 
     
       1. A method for generating synthetic full backups, the method comprising:
 performing a verification that a previous backup of source data stored in a data domain (DD) is a failed synthetic full backup (SFB); 
 obtaining, based on the verification and based on a user initiated backup request to perform another backup of the source data to the DD, a latest snapshot of the source data, wherein the latest snapshot of the source data is a most recently taken successful snapshot of the source data before the failed SFB, wherein a source node is configured as a Hadoop cluster and stores the source data; 
 obtaining, based on the verification, a prior snapshot of the source data, wherein the prior snapshot is created before the latest snapshot and used to perform the failed SFB; 
 generating a snapshot difference report using the latest snapshot and the prior snapshot, wherein the snapshot difference report comprises a delete list, a rename list, and a copy list comprising a list of data items; 
 making a first determination that a copy started file (CSF) does not exist in the previous backup, wherein the non-existence of the CSF indicates that neither a delete operation and a rename operation was performed on the previous backup; 
 making, after the first determination, a second determination, using the copy list, that a first portion of the data items in the copy list exists in the previous backup, and a second portion of the data items does not exist in the previous backup; 
 performing, based on the second determination and using the snapshot difference report, a copy operation on a copy of the previous backup to copy the second portion of the data items to the DD to obtain a SFB; and 
 updating, upon obtaining the SFB, permissions and attributes associated with files and/or folders of the source data included in the SFB to verify a status of the SFB during a next synthetic full backup cycle. 
 
     
     
       2. The method of  claim 1 , further comprising:
 based on the first determination:
 performing a delete operation using the delete list; and 
 performing a rename operation using the rename list. 
 
 
     
     
       3. The method of  claim 1 , wherein performing the verification comprises:
 obtaining a snapshot name associated with the previous backup; and 
 making a third determination that the snapshot name does not correspond to the latest snapshot. 
 
     
     
       4. The method of  claim 1 , wherein the source data is a Hadoop Distributed File System (HDFS). 
     
     
       5. The method of  claim 1 , wherein the previous backup is one selected from a group consisting of an initial full backup and a second synthetic full backup. 
     
     
       6. The method of  claim 2 , further comprising:
 ignoring, during the delete operation, a delete operation error; and 
 ignoring, during the rename operation, a rename operation error. 
 
     
     
       7. A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method for generating synthetic full backups, the method comprising:
 performing a verification that a previous backup of source data stored in a data domain (DD) is a failed synthetic full backup (SFB); 
 obtaining, based on the verification and based on a user initiated backup request to perform another backup of the source data to the DD, a latest snapshot of the source data, wherein the latest snapshot of the source data is a most recently taken successful snapshot of the source data before the failed SFB, wherein a source node is configured as a Hadoop cluster and stores the source data; 
 obtaining, based on the verification, a prior snapshot of the source data, wherein the prior snapshot is created before the latest snapshot and used to perform the failed SFB; 
 generating a snapshot difference report using the latest snapshot and the prior snapshot, wherein the snapshot difference report comprises a delete list, a rename list, and a copy list comprising a list of data items; 
 making a first determination that a copy started file (CSF) does not exist in the previous backup, wherein the non-existence of the CSF indicates that neither a delete operation and a rename operation was performed on the previous backup; 
 making, after the first determination, a second determination, using the copy list, that a first portion of the data items in the copy list exists in the previous backup, and a second portion of the data items does not exist in the previous backup; 
 performing, based on the second determination and using the snapshot difference report, a copy operation on a copy of the previous backup to copy the second portion of the data items to the DD to obtain a SFB; and 
 updating, upon obtaining the SFB, permissions and attributes associated with assets files and/or folders of the source data included in the SFB to verify a status of the SFB during a next synthetic full backup cycle. 
 
     
     
       8. The non-transitory computer readable medium of  claim 7 , further comprising:
 based on the first determination:
 performing a delete operation using the delete list; and 
 performing a rename operation using the rename list. 
 
 
     
     
       9. The non-transitory computer readable medium of  claim 7 , wherein performing the verification comprises:
 obtaining a snapshot name associated with the previous backup; and 
 making a third determination that the snapshot name does not correspond to the latest snapshot. 
 
     
     
       10. The non-transitory computer readable medium of  claim 7 , wherein the source data is a Hadoop Distributed File System (HDFS). 
     
     
       11. The non-transitory computer readable medium of  claim 7 , wherein the previous backup is one selected from a group consisting of an initial full backup and a second synthetic full backup. 
     
     
       12. The non-transitory computer readable medium of  claim 8 , further comprising:
 ignoring, during the delete operation, a delete operation error; and 
 ignoring, during the rename operation, a rename operation error. 
 
     
     
       13. A system for generating synthetic full backups, the system comprising:
 a processor comprising circuitry; 
 memory; and 
 a source node operatively connected to a data domain (DD), executing on the processor and using the memory, and configured to:
 perform a verification that a previous backup of source data stored in the DD ais a failed synthetic full backup (SFB); 
 obtain, based on the verification and based on a user initiated backup request to perform another backup of the source data to the DD, a latest snapshot of the source data, wherein the latest snapshot of the source data is a most recently taken successful snapshot of the source data before the failed SFB, wherein a source node is configured as a Hadoop cluster and stores the source data; 
 obtain, based on the verification, a prior snapshot of the source data, wherein the prior snapshot is created before the latest snapshot and used to perform the failed SFB; 
 generate a snapshot difference report using the latest snapshot and the prior snapshot, wherein the snapshot difference report comprises a delete list, a rename list, and a copy list comprising a list of data items; 
 make a first determination that a copy started file (CSF) does not exist in the previous backup, wherein the non-existence of the CSF indicates that neither a delete operation and a rename operation was performed on the previous backup; 
 make, after the first determination, a second determination, using the copy list, that a first portion of the data items in the copy list exists in the previous backup, and a second portion of the data items does not exist in the previous backup; 
 perform, based on the second determination and using the snapshot difference report, a copy operation on a copy of the previous backup to copy the second portion of the data items to the DD to obtain a SFB; and 
 update, upon obtaining the SFB, permissions and attributes associated with files and/or folders of the source data included in the SFB to verify a status of the SFB during a next synthetic full backup cycle. 
 
 
     
     
       14. The system of  claim 13 , wherein the source node is further configured to:
 based on the first determination:
 perform a delete operation using the delete list; and 
 perform a rename operation using the rename list. 
 
 
     
     
       15. The system of  claim 13 , wherein to perform the verification, the source node is further configured to:
 obtain a snapshot name associated with the previous backup; and 
 make a third determination that the snapshot name does not correspond to the latest snapshot. 
 
     
     
       16. The system of  claim 13 , wherein the source data is a Hadoop Distributed File System (HDFS). 
     
     
       17. The system of  claim 14 , wherein the source node is further configured to:
 ignore, during the delete operation, a delete operation error; and 
 ignore, during the rename operation, a rename operation error.