Metadata management method, system and medium

Abstract

A metadata management method, system and medium are provided. All directories and files in a file system are organized in a tree structure, and all directories and subfiles under a single directory are saved as a whole on a continuous address space of an underlying storage device in the order of creation in sequence. When metadata is updated, update operations on all subdirectories/subfiles under a directory are appended to the tail of the directory in the form of log in order, so that each metadata operation triggers only one IO request. In the process of metadata retrieval, parsing a subdirectory under a parent director requires reading all data on the continuous address space corresponding to the parent directory, and then searching for a match by means of multiple acceleration methods.

What is claimed is: 
     
       1. A metadata management method executed by a metadata management system comprising a computer system, wherein the computer system comprises a processor and a non-transitory storage medium, wherein implementation steps for updating metadata in the metadata management method comprises:
 1) organizing, by the processor, all directories and files in a file system of the non-transitory storage medium in a tree structure, and saving, by the processor, all directories and subfiles under a single directory as a whole on a continuous address space of an underlying non-transitory storage medium in an order of creation in sequence; and 
 2) updating, by the processor, all subdirectories/subfiles under the single directory in a form of a log when updating the metadata of the file system, and appending, by the processor, all metadata updates to a tail of the continuous address space corresponding to the single directory in an order of occurrence so that each metadata operation triggers only one IO request to the non-transitory storage medium, 
 wherein implementation steps for metadata retrieval comprise: 
 S1) organizing, by the processor, all the directories and the files in the file system of the non-transitory storage medium in the tree structure, and saving, by the processor, all the directories and the subfiles under the single directory as the whole in the continuous address space of the underlying non-transitory storage medium in the order of creation in sequence; and 
 S2) when the processor performs file system metadata retrieval, the processor parses a directory path of the non-transitory storage medium layer by layer from a first component to a last component in sequence and finally obtains a directory or a file to be accessed in the non-transitory storage medium; and when the processor parses a subdirectory under a parent directory, the processor reads all data on the continuous address space of the non-transitory storage medium corresponding to the parent directory, and then searches a target subdirectory or a subfile on read data of the non-transitory storage medium until a match is successful, 
 wherein the metadata retrieval is optimized according to a principle of locality, 
 wherein parsing the subdirectory under the parent directory comprises following detailed steps: 
 S2.1) the processor receives a metadata retrieval request from an application, and the metadata retrieval request being to retrieve an existence of the target directory or file from the parent directory; 
 S2.2) the processor obtains related information of all the subdirectories/subfiles under the parent directory into the non-transitory storage medium through one read operation, and going to step S2.3); 
 S2.3) the processor determines a size of the parent directory to be parsed, and the processor executes step S2.4) if the total number of the subdirectories/subfiles in the parent directory is less than a preset first threshold TH 1 ; or the processor executes step S2.5) if the total number of the subdirectories/subfiles in the parent directory is between the preset first threshold TH 1  and a preset second threshold TH 2 ; or the processor executes step S2.6) if the total number of the subdirectories/subfiles in the parent directory is greater than the preset second threshold TH 2 ; 
 S2.4) the processor uses multi-threading technology to search the target subdirectory or subfile in the parent directory, and going to step S2.7); 
 S2.5) the processor uses a vector instruction to search the target subdirectory or subfile in the parent directory, and going to step S2.7); 
 S2.6) the processor uses many-core acceleration technology to search the target subdirectory or subfile in the parent directory, and going to step S2.7); and 
 S2.7) the processor succeeds in the retrieval if the target subdirectory or subfile is found in the parent directory of the non-transitory storage medium; or the processor fails in the retrieval, and returns a retrieval result to an upper application, 
 wherein a linear search process is performed when searching the target subdirectory or the subfile in the parent directory because no index is established for all the subdirectories/subfiles under the same parent directory to save IO operation, wherein the multi-threading technology, the vector instruction and the many-core acceleration technology shorten a computation time of the linear search process. 
 
     
     
       2. The metadata management method according to  claim 1 , wherein detailed steps of the step 2) comprise:
 2.1) receiving a metadata update operation by the file system from an application; 
 2.2) the processor determines a type of the metadata update operation, and the processor executes step 2.3) if the type is creating a new directory or file, or the processor executes step 2.4) if the type is updating an existing directory or file, or the processor executes step 2.5) if the type is deleting an existing directory or file; 
 2.3) the processor generates a file/directory creation log, and going to step 2.6); 
 2.4) the processor generates a directory/file update log, and going to step 2.6); 
 2.5) the processor generates a directory/file deletion log, and going to step 2.6); 
 2.6) the processor appends the generated log to the tail of the continuous address space, and going to step 2.7); and 
 2.7) the processor persists the tail of the continuous address space to the non-transitory storage medium. 
 
     
     
       3. The metadata management method according to  claim 1 , wherein the processor searches the target subdirectory or the subfile on the read data by a reverse processing from the tail of the continuous address space to a head of the continuous address space, and the processor terminates the retrieval process and immediately returning to the corresponding metadata once the subdirectory/subfile to be searched is retrieved. 
     
     
       4. The metadata management method according to  claim 1 , wherein, in at least one of the step S2.4), step S2.5) and Step S2.6), when the processor searches the target subdirectory or subfile in the parent directory, the processor searches by reverse processing from the tail of the continuous address space to the head of the continuous address space. 
     
     
       5. A metadata management system, comprising a computer system, wherein, the computer system programmed to execute the steps of the metadata management method of  claim 1 , or a computer program programmed to execute the metadata management method of  claim 1  is stored on a non-transitory storage medium of the computer system. 
     
     
       6. A computer-readable non-transitory storage medium, wherein, on which a computer program programmed to execute the metadata management method of  claim 1  is stored. 
     
     
       7. A metadata management system, comprising a computer system, wherein, the computer system programmed to execute the steps of the metadata management method of  claim 2 , or a computer program programmed to execute the metadata management method of  claim 2  is stored on a non-transitory storage medium of the computer system. 
     
     
       8. A metadata management system, comprising a computer system, wherein, the computer system programmed to execute the steps of the metadata management method of  claim 3 , or a computer program programmed to execute the metadata management method of  claim 3  is stored on a non-transitory storage medium of the computer system. 
     
     
       9. A metadata management system, comprising a computer system, wherein, the computer system programmed to execute the steps of the metadata management method of  claim 4 , or a computer program programmed to execute the metadata management method of  claim 4  is stored on a non-transitory storage medium of the computer system. 
     
     
       10. A computer-readable storage non-transitory medium, wherein, on which a computer program programmed to execute the metadata management method of  claim 2  is stored. 
     
     
       11. A metadata management method executed by a metadata management system comprising a computer system, wherein the computer system comprises a processor and a non-transitory storage medium, wherein implementation steps for metadata retrieval in the metadata management method comprises:
 S1) organizing, by the processor, all directories and files in a file system of the non-transitory storage medium in a tree structure, and saving, by the processor, all directories and subfiles under a single directory as a whole in a continuous address space of an underlying non-transitory storage medium in an order of creation in sequence; and 
 S2) when the processor performs file system metadata retrieval, the processor parses a directory path of the non-transitory storage medium layer by layer from a first component to a last component in sequence and finally obtaining a directory or a file to be accessed in the non-transitory storage medium; and when the processor parses a subdirectory under a parent directory, the processor reads all data on the continuous address space of the non-transitory storage medium, and then searches a target subdirectory or a subfile on read data until a match is successful, 
 wherein the metadata retrieval is optimized according to a principle of locality, 
 wherein, parsing the subdirectory under the parent directory comprises following detailed steps: 
 S2.1) the processor receives a metadata retrieval request from an application, and the metadata retrieval request being to retrieve an existence of the target directory or file from the parent directory; 
 S2.2) the processor obtains related information of all the subdirectories/subfiles under the parent directory into a memory through one read operation, and going to step S2.3); 
 S2.3) the processor determines a size of the parent directory to be parsed, and the processor executes step S2.4) if the total number of the subdirectories/subfiles in the parent directory is less than a preset first threshold TH 1 ; or the processor executes step S2.5) if the total number of the subdirectories/subfiles in the parent directory is between the preset first threshold TH 1  and a preset second threshold TH 2 ; or the processor executes step S2.6) if the total number of the subdirectories/subfiles in the parent directory is greater than the preset second threshold TH 2 ; 
 S2.4) the processor uses multi-threading technology to search the target subdirectory or subfile in the parent directory, and going to step S2.7); 
 S2.5) the processor uses a vector instruction to search the target subdirectory or subfile in the parent directory, and going to step S2.7); 
 S2.6) the processor uses many-core acceleration technology to search the target subdirectory or subfile in the parent directory, and going to step S2.7); and 
 S2.7) the processor succeeds in the retrieval if the target subdirectory or subfile is found in the parent directory of the non-transitory storage medium; or the processor fails in the retrieval, and returning a retrieval result to an upper application, 
 wherein a linear search process is performed when searching the target subdirectory or the subfile in the parent directory because no index is established for all the subdirectories/subfiles under the same parent directory to save IO operation, wherein the multi-threading technology, the vector instruction and the many-core acceleration technology shorten a computation time of the linear search process. 
 
     
     
       12. A metadata management system, comprising a computer system, wherein, the computer system programmed to execute the steps of the metadata management method of  claim 11 , or a computer program programmed to execute the metadata management method of  claim 11  is stored on a non-transitory storage medium of the computer system.