Parallel Filesystem Traversal For Transparent Mirroring of Directories and Files

Abstract

A system and method for parallel file system traversal using multiple job executors is disclosed. The system includes a pool of job executors, a job queue, and a trigger tracker. An object, representative of a node in the filesystem, is added (i.e., pushed) to the job queue for processing by an job executor. The job queue assigns (i.e., pops) objects to job executors in accordance to a LIFO (Last In First Out) ordering. Then the job executor performs an action such as copy. In one embodiment, the trigger tracker follows the processing of a child nodes to a particular child node. Thus, the filesystem is being traversed by several job executors at the same time.

What is claimed is: 
     
         1 . A method comprising:
 selecting one or more sets of objects, each object in a set being representative of a child node stemming from at least one of one or more common parent nodes in the file system;   assigning one or more objects from the one of the one or more sets of objects to a plurality of job executors, wherein assignment of an object is prevented before completing operations on each object positioned in relation to a barrier; and   performing parallel actions on the assigned objects, wherein parallel actions include at least one of replication and migration of data characterizing an object,   wherein the plurality of job executors are configured to perform pre-processing on an assigned object when the assigned object has child nodes that have not yet been processed, and to perform post-processing on an assigned object when the assigned object has child nodes that have already been processed.   
     
     
         2 . The method of  claim 1 , further comprising:
 adding the selected one or more sets of objects to a job queue; and   incrementing a respective one of one or more counts associated with an object in accordance with a number of child nodes associated with a corresponding one of the one or more common parent nodes represented by the object.   
     
     
         3 . The method of  claim 2 , further comprising inserting a barrier between the objects in the job queue, the barrier preventing assignment of an object before the barrier prior to completing operations on each object after the barrier. 
     
     
         4 . The method of  claim 1 , wherein completing operations on an object in the job queue comprises performing parallel actions on objects representative of child nodes stemming from another object. 
     
     
         5 . The method of  claim 1 , wherein assigning objects comprises:
 detecting that each of the plurality of job executors is awaiting an assignment from the job queue; and   assigning the objects from the job queue to the plurality of job executors before adding additional sets of objects to the job queue.   
     
     
         6 . The method of  claim 1 , wherein assigning objects comprises:
 detecting that each of the plurality of job executors has a set of objects to add to the job queue; and   adding each set to the job queue before assigning additional objects.   
     
     
         7 . The method of  claim 1 , further comprising requesting information for generating a set of objects. 
     
     
         8 . The method of  claim 1 , wherein the parallel actions are part of one of a namespace replication or a namespace migration. 
     
     
         9 . The method of  claim 1 , wherein the parallel actions are part of at least one of a snapshot application and a data replication. 
     
     
         10 . The method of  claim 1 , wherein the plurality of job executors comprises a plurality of threads. 
     
     
         11 . A system for parallel traversal of a file system with a plurality of job executors, the system comprising:
 a processor;   a non-transitory memory storing instructions that when executed by a processor cause a system to operate to:   select one or more sets of objects, each object in a set being representative of a child node stemming from at least one of one or more common parent nodes in the file system;   assign one or more objects from the one of the one or more sets of objects to a plurality of job executors, wherein assignment of an object is prevented before completing operations on each object positioned in relation to a barrier; and   perform parallel actions on the assigned objects, wherein parallel actions include at least one of replication and migration of data characterizing an object,   wherein the plurality of job executors are configured to perform pre-processing on an assigned object when the assigned object has child nodes that have not yet been processed, and to perform post-processing on an assigned object when the assigned object has child nodes that have already been processed   
     
     
         12 . The system of  claim 11 , wherein the non-transitory memory further comprises instructions when executed by the processor cause the system to operate to:
 add the selected one or more sets of objects to a job queue; and   increment a respective one of one or more counts associated with an object in accordance with a number of child nodes associated with a corresponding one of the one or more common parent nodes represented by the object.   
     
     
         13 . The system of  claim 12 , wherein the non-transitory memory further comprises instructions when executed by the processor cause the system to operate to insert a barrier between the objects in the job queue, the barrier preventing assignment of an object before the barrier prior to completing operations on each object after the barrier. 
     
     
         14 . The system of  claim 11 , wherein completing operations on an object in the job queue comprises performing parallel actions on objects representative of child nodes stemming from another object. 
     
     
         15 . The system of  claim 11 , wherein assigning objects comprises:
 detecting that each of the plurality of job executors is awaiting an assignment from the job queue; and   assigning the objects from the job queue to the plurality of job executors before adding additional sets of objects to the job queue   
     
     
         16 . The system of  claim 11 , wherein assigning objects comprises:
 detecting that each of the plurality of job executors has a set of objects to add to the job queue; and   adding each set to the job queue before assigning additional objects.   
     
     
         17 . The system of  claim 11 , wherein the non-transitory memory further comprises instructions when executed by the processor cause the system to operate to request information for generating a set of objects. 
     
     
         18 . The system of  claim 11 , wherein the parallel actions are part of one of a namespace replication or a namespace migration. 
     
     
         19 . The system of  claim 11 , wherein the plurality of job executors comprises a plurality of threads. 
     
     
         20 . The system of  claim 11 , wherein the parallel actions are part of at least one of a snapshot application and a data replication