Data storage system for providing low latency search query responses

Abstract

A computer-implemented method of providing low-latency responses to queries using a distributed storage system comprising a plurality of distributed nodes is disclosed. The method comprises generating a plurality of database views, including a database view joining first digital data and second digital data, the first digital data being stored in a first digital data repository of the distributed storage system using a first storage format, the second digital data being stored in a second digital data repository using a second storage format different from the first storage format; storing the plurality of database views in a third digital data repository using a column-oriented storage format; storing metadata for the plurality of database views, including schema information for the database views and location information of one or more distributed nodes storing data corresponding to the database views; loading the metadata into a search system configured to receive a search query and return a result set for the search query based the metadata.

What is claimed is: 
     
         1 . A computer-implemented method of providing low-latency responses to queries using a distributed storage system comprising a plurality of distributed nodes, comprising:
 generating, by a processor, a plurality of database views, including a database view joining first digital data and second digital data,   the first digital data being stored in a first digital data repository of the distributed storage system using a first storage format,   the second digital data being stored in a second digital data repository of the distributed storage system using a second storage format different from the first storage format;   storing, by the processor, the plurality of database views in a third digital data repository of the distributed storage system using a column-oriented storage format;   storing metadata for the plurality of database views, the metadata including schema information for the database views and location information of one or more distributed nodes of the plurality of distribute nodes storing data corresponding to the database views;   loading the metadata into a search system configured to receive a search query and return a result set for the search query based the metadata.   
     
     
         2 . The computer-implemented method of  claim 1 , further comprising:
 receiving a search query;   executing the search query based on the metadata to generate a result set;   transmitting the result set in response to the search query.   
     
     
         3 . The computer-implemented method of  claim 2 , the executing comprising selecting one or more database views of the plurality of database views and retrieving one or more columns from the one or more database views based on the search query and the metadata. 
     
     
         4 . The computer-implemented method of  claim 1 ,
 the first digital data corresponding to a first risk type associated with a first schema,   the second digital data corresponding to a second risk type associated with a second schema,   the metadata for the database view including schema information based on the first schema and the second schema.   
     
     
         5 . The computer-implemented method of  claim 1 , the first storage format being a first column-oriented storage format and or the second storage format being a second column-oriented storage format. 
     
     
         6 . The computer-implemented method of  claim 5 , further comprising selecting the first column-oriented storage format or the second column-oriented storage format based on access or update patterns of digital data respectively stored in the first digital data repository or the second digital data repository. 
     
     
         7 . The computer-implemented method of  claim 5 ,
 the first digital data requiring an amount of update exceeding a threshold, and the first column-oriented storage format being a relational data format,   the second digital data requiring an amount of update not exceeding the threshold, and the second column-oriented storage format comprising data files associated with schema files,   the column-oriented storage format used in the third digital data repository being the second column-oriented storage format.   
     
     
         8 . The computer-implemented method of  claim 1 , the database view being generated based on an example query, user input indicating a particular pre-computed view, or a configuration file. 
     
     
         9 . The computer-implemented method of  claim 1 , the generating being automatically performed when new digital data is loaded into or existing data changes in the first digital data repository or the second digital data repository. 
     
     
         10 . The computer-implemented method of  claim 1 , the first digital data repository or the second digital data repository being a wide column store. 
     
     
         11 . One or more non-transitory computer-readable storage media storing instructions which when executed cause one or more processors to perform a method of providing low-latency responses to queries using a distributed storage system comprising a plurality of distributed nodes, the method comprising:
 generating a plurality of database views, including a database view joining first digital data and second digital data,   the first digital data being stored in a first digital data repository of the distributed storage system using a first storage format,   the second digital data being stored in a second digital data repository of the distributed storage system using a second storage format different from the first storage format;   storing the plurality of database views in a third digital data repository of the distributed storage system using a column-oriented storage format;   storing metadata for the plurality of database views, the metadata including schema information for the database views and location information of one or more distributed nodes of the plurality of distribute nodes storing data corresponding to the database views;   loading the metadata into a search system configured to receive a search query and return a result set for the search query based the metadata.   
     
     
         12 . The one or more non-transitory computer-readable storage media of  claim 11 , the method further comprising:
 receiving a search query;   executing the search query based on the metadata to generate a result set;   transmitting the result set in response to the search query.   
     
     
         13 . The one or more non-transitory computer-readable storage media of  claim 12 , the executing comprising selecting one or more database views of the plurality of database views and retrieving one or more columns from the one or more database views based on the search query and the metadata. 
     
     
         14 . The one or more non-transitory computer-readable storage media of  claim 11 ,
 the first digital data corresponding to a first risk type associated with a first schema,   the second digital data corresponding to a second risk type associated with a second schema,   the metadata for the database view including schema information based on the first schema and the second schema.   
     
     
         15 . The one or more non-transitory computer-readable storage media of  claim 11 , the first storage format being a first column-oriented storage format and or the second storage format being a second column-oriented storage format. 
     
     
         16 . The one or more non-transitory computer-readable storage media of  claim 15 , the method further comprising selecting the first column-oriented storage format or the second column-oriented storage format based on access or update patterns of digital data respectively stored in the first digital data repository or the second digital data repository. 
     
     
         17 . The one or more non-transitory computer-readable storage media of  claim 15 ,
 the first digital data requiring an amount of update exceeding a threshold, and the first column-oriented storage format being a relational data format,   the second digital data requiring an amount of update not exceeding the threshold, and the second column-oriented storage format comprising data files associated with schema files,   the column-oriented storage format used in the third digital data repository being the second column-oriented storage format.   
     
     
         18 . The one or more non-transitory computer-readable storage media of  claim 11 , the database view being generated based on an example query, user input indicating a particular pre-computed view, or a configuration file. 
     
     
         19 . The one or more non-transitory computer-readable storage media of  claim 11 , the generating being automatically performed when new digital data is loaded into or existing data changes in the first digital data repository or the second digital data repository. 
     
     
         20 . The one or more non-transitory computer-readable storage media of  claim 11 , the first digital data repository or the second digital data repository being a wide column store.