Data processing method and apparatus, computer device and storage medium

Abstract

The present disclosure provides a data processing method and apparatus, a computer device and a storage medium. The method comprises: receiving a query instruction for data to be queried stored in a data file system; in response to the query instruction instructing to perform online query on the data, determining index information corresponding to the data, and querying the data in the data file system on the basis of the index information; and in response to the query instruction instructing to perform offline query on the data, determining address information of the data in the data file system on the basis of a keyword of the data, and querying the data in the data file system on the basis of the address information.

We claim: 
     
         1 . A method for data processing, comprising:
 receiving a query instruction for data to be queried stored in a data file system, wherein the data file system is a common database storing data for online queries and offline queries;   determining whether the query instruction indicates an online query or an office query;   in response to the query instruction indicating the online query, determining index information corresponding to the data to be queried and querying, based on the index information, the data to be queried in the data file system, wherein the index information is represented in a key-value pair, a key in the key-value pair corresponds to a keyword of the data to be queried, and the value in the key-value pair corresponds to address information of the data to be queried in the data file system; and   in response to the query instruction indicating the offline query, determining, based on keywords of the data to be queried and a hive table, address information of the data to be queried in the data file system and querying, based on the address information, the data to be queried in the data file system, wherein the hive table comprises the keywords corresponding to the data to be queried and address information of the data to be queried in the data file system.   
     
     
         2 . The method of  claim 1 , wherein determining, based on keywords of the data to be queried, address information of the data to be queried in the data file system comprises:
 determining, based on the keywords of the data to be queried, address information of the data to be queried in an offline data table corresponding top the data file system; and   the offline data table includes the keywords corresponding to the data to be queried, and the address information of the data to be queried in the data file system.   
     
     
         3 . The method of  claim 1 , wherein determining index information corresponding to the data to be queried comprises:
 determining, based on the keywords of the data to be queried, index information corresponding to the data to be queried in a data index base corresponding to the data file system,   wherein the data index base includes the index information of the data to be queried, the index information of the data to be queried includes the keywords of the data to be queried, and address information of the data to be queried in the data file system.   
     
     
         4 . The method of  claim 3 , wherein the data file system includes a plurality of storage data containing the data to be queried and each of the storage data has corresponding index information;
 before determining, based on the keywords of the data to be queried, the index information corresponding to the data to be queried in the data index base, the method also comprises:   grouping a plurality of pieces of the index information and updating, based on a plurality of pieces of the index information groups obtained, the data index base; and   wherein determining, based on the keywords of the data to be queried, the index information corresponding to the data to be queried in the data index base comprises:   determining, based on the keywords, a target index information group from the plurality of pieces of index information groups and an index position of the index information corresponding to the data to be queried in the target index information group; and   determining, based on the index position, the index information of the data to be queried from the index information group.   
     
     
         5 . The method of  claim 4 , wherein obtaining the index information groups by grouping a plurality of pieces of the index information comprises:
 determining grouping keywords for the index information groups;   determining, based on the grouping keywords, target index information from a plurality of pieces of the index information, wherein the grouping keywords include keywords of the target index information; and   grouping the target index information to obtain the index information groups.   
     
     
         6 . The method of  claim 1 , wherein querying, based on the index information, the data to be queried in the data file system comprises:
 querying, based on address information in the index information, the data to be queried in the data file system; and   the address information represents a relative storage position of the data to be queried in a target data block of the data file system, the target data block being located under a target file path in the data file system.   
     
     
         7 . The method of  claim 1 , wherein the keywords include at least one of: object information, type information, corresponding query scope under the type information and a data identifier corresponding to the data to be queried. 
     
     
         8 . A computer device, comprising: a processor and a memory stored with machine-readable instructions executable by the processor; wherein the machine-readable instructions, when executed by the processor, cause the processor to perform a method comprising:
 receiving a query instruction for data to be queried stored in a data file system, wherein the data file system is a common database storing data for online queries and offline queries;   determining whether the query instruction indicates an online query or an office query;   in response to the query instruction indicating the online query, determining index information corresponding to the data to be queried and querying, based on the index information, the data to be queried in the data file system, wherein the index information is represented in a key-value pair, a key in the key-value pair corresponds to a keyword of the data to be queried, and the value in the key-value pair corresponds to address information of the data to be queried in the data file system; and   in response to the query instruction indicating the offline query, determining, based on keywords of the data to be queried and a hive table, address information of the data to be queried in the data file system and querying, based on the address information, the data to be queried in the data file system, wherein the hive table comprises the keywords corresponding to the data to be queried and address information of the data to be queried in the data file system.   
     
     
         9 . A non-transitory computer-readable storage medium stored thereon with computer programs which, when operated by a computer device, cause the computer device to perform a method comprising
 receiving a query instruction for data to be queried stored in a data file system, wherein the data file system is a common database storing data for online queries and offline queries;   determining whether the query instruction indicates an online query or an office query;   in response to the query instruction indicating the online query, determining index information corresponding to the data to be queried and querying, based on the index information, the data to be queried in the data file system, wherein the index information is represented in a key-value pair, a key in the key-value pair corresponds to a keyword of the data to be queried, and the value in the key-value pair corresponds to address information of the data to be queried in the data file system; and   in response to the query instruction indicating the offline query, determining, based on keywords of the data to be queried and a hive table, address information of the data to be queried in the data file system and querying, based on the address information, the data to be queried in the data file system, wherein the hive table comprises the keywords corresponding to the data to be queried and address information of the data to be queried in the data file system.   
     
     
         10 . The computer device of  claim 8 , wherein determining, based on keywords of the data to be queried, address information of the data to be queried in the data file system comprises:
 determining, based on the keywords of the data to be queried, address information of the data to be queried in an offline data table corresponding top the data file system; and   the offline data table includes the keywords corresponding to the data to be queried, and the address information of the data to be queried in the data file system.   
     
     
         11 . The computer device of  claim 8 , wherein determining index information corresponding to the data to be queried comprises:
 determining, based on the keywords of the data to be queried, index information corresponding to the data to be queried in a data index base corresponding to the data file system,   wherein the data index base includes the index information of the data to be queried, the index information of the data to be queried includes the keywords of the data to be queried, and address information of the data to be queried in the data file system.   
     
     
         12 . The computer device of  claim 11 , wherein the data file system includes a plurality of storage data containing the data to be queried and each of the storage data has corresponding index information;
 before determining, based on the keywords of the data to be queried, the index information corresponding to the data to be queried in the data index base, the method also comprises:   grouping a plurality of pieces of the index information and updating, based on a plurality of pieces of the index information groups obtained, the data index base; and   wherein determining, based on the keywords of the data to be queried, the index information corresponding to the data to be queried in the data index base comprises:   determining, based on the keywords, a target index information group from the plurality of pieces of index information groups and an index position of the index information corresponding to the data to be queried in the target index information group; and   determining, based on the index position, the index information of the data to be queried from the index information group.   
     
     
         13 . The computer device of  claim 12 , wherein obtaining the index information groups by grouping a plurality of pieces of the index information comprises:
 determining grouping keywords for the index information groups;   determining, based on the grouping keywords, target index information from a plurality of pieces of the index information, wherein the grouping keywords include keywords of the target index information; and   grouping the target index information to obtain the index information groups.   
     
     
         14 . The computer device of  claim 8 , wherein querying, based on the index information, the data to be queried in the data file system comprises:
 querying, based on address information in the index information, the data to be queried in the data file system; and   the address information represents a relative storage position of the data to be queried in a target data block of the data file system, the target data block being located under a target file path in the data file system.   
     
     
         15 . The computer device of  claim 8 , wherein the keywords include at least one of: object information, type information, corresponding query scope under the type information and a data identifier corresponding to the data to be queried. 
     
     
         16 . The non-transitory computer-readable storage medium of  claim 9 , wherein determining, based on keywords of the data to be queried, address information of the data to be queried in the data file system comprises:
 determining, based on the keywords of the data to be queried, address information of the data to be queried in an offline data table corresponding top the data file system; and   the offline data table includes the keywords corresponding to the data to be queried, and the address information of the data to be queried in the data file system.   
     
     
         17 . The non-transitory computer-readable storage medium of  claim 9 , wherein determining index information corresponding to the data to be queried comprises:
 determining, based on the keywords of the data to be queried, index information corresponding to the data to be queried in a data index base corresponding to the data file system,   wherein the data index base includes the index information of the data to be queried, the index information of the data to be queried includes the keywords of the data to be queried, and address information of the data to be queried in the data file system.   
     
     
         18 . The non-transitory computer-readable storage medium of  claim 17 , wherein the data file system includes a plurality of storage data containing the data to be queried and each of the storage data has corresponding index information;
 before determining, based on the keywords of the data to be queried, the index information corresponding to the data to be queried in the data index base, the method also comprises:   grouping a plurality of pieces of the index information and updating, based on a plurality of pieces of the index information groups obtained, the data index base; and   wherein determining, based on the keywords of the data to be queried, the index information corresponding to the data to be queried in the data index base comprises:   determining, based on the keywords, a target index information group from the plurality of pieces of index information groups and an index position of the index information corresponding to the data to be queried in the target index information group; and   determining, based on the index position, the index information of the data to be queried from the index information group.   
     
     
         19 . The non-transitory computer-readable storage medium of  claim 18 , wherein obtaining the index information groups by grouping a plurality of pieces of the index information comprises:
 determining grouping keywords for the index information groups;   determining, based on the grouping keywords, target index information from a plurality of pieces of the index information, wherein the grouping keywords include keywords of the target index information; and   grouping the target index information to obtain the index information groups.   
     
     
         20 . The non-transitory computer-readable storage medium of  claim 9 , wherein querying, based on the index information, the data to be queried in the data file system comprises:
 querying, based on address information in the index information, the data to be queried in the data file system; and   the address information represents a relative storage position of the data to be queried in a target data block of the data file system, the target data block being located under a target file path in the data file system.