System for organizing and fast searching massive amounts of data

Abstract

A system to collect and store in a special data structure arranged for rapid searching massive amounts of data. Performance metric data is one example. The performance metric data is recorded in time-series measurements, converted into unicode, and arranged into a special data structure having one directory for every day which stores all the metric data collected that day. The performance metric data is collected by one or more probes running on machines about which data is being collected. The performance metric data is compressed prior to transmission to a server over any data path. The data structure at the server where analysis is done has a subdirectory for every resource type. Each subdirectory contains text files of performance metric data values measured for attributes in a group of attributes to which said text file is dedicated. Each attribute has its own section and the performance metric data values are recorded in time series as unicode hex numbers as a comma delimited list. Analysis of the performance metric data is done using regular expressions. For speed, cache memory is used. Performance metric data outside the start time and end time named by the user on a query screen is eliminated before the regular expression is applied.

What is claimed is: 
     
         1 . A process for performing multiple searches of text configuration data and/or Unicode encoded performance data stored in a non relational database file system along multiple traversal paths through muliple levels of nested searches, comprising:
 executing a first search on a first level of nesting to find all instances of a first resource type specified in said first search which have performance or configuration date which meet a filter criteria given in a regular expression in said first search, wherein said filter criteria may be no filter criteria at all;   executing a second search on a second level of nesting to find all instances of a second resource type specified in said second search which are related to instances of said first resource type returned by said first search which have configuration or performance data which meet a filter criteria given in a regular expression given in said second search; and   executing a third search on a second level of nesting to find all instances of a third resource type specified in said third search which are related to instances of said first resource type returned by said first search which have configuration or performance data which meet a filter criteria given in a regular expression given in said third search; and   returning at least the identies of all instances which met said filter criteria of said second and third searches,   
     
     
         2 . The process of  claim 1  further comprising returning data which met said filter criteria of said second and third searches in addition to the identities of all instances which met third searches. 
     
     
         3 . The process of  claim 1  wherein said second and third searches are carried out concurrently, 
     
     
         4 . A computer apparatus comprising:
 a display,   a memory storing one or more computer programs;   an input device;   a data store storing configuration data and Unicode characters mapped from numerical values of performance metric data defining values over time of one or more attributes of one or more instances of one or more resource types in a system being monitored, said Unicode characters stored in one or more files of a non-relational database file system;   a programmable processor coupled to said memory and capable of being controlled by said computer program(s) stored therein, and coupled to said display, said input device, and said data store;   and wherein said one or more computer programs are structured to control said programmable processor to perform the following process to implement a tree search of data in said data store:   A) receiving a search query defining a relevant interval and first search on a first level of nesting and second and third searches on a second level of nesting, each said search specifying an attribute and having a filter criteria expressed in the form of a regular expression, at least a first search on each level of nesting specifying a resource type; and   B) executing said first search and passing results to each of said second and third searches, and   C) executing each of said second and third searches and passing results to a results cache.   
     
     
         5 . The apparatus of  claim 4  wherein each of said second and third searches are for resource types which related to the resource type of said first search. 
     
     
         6 . The apparatus of  claim 4  wherein said second and third searches pass both instances of resource types which passed said second and third searches to said results cache but also pass data of said instances which passed said second and third searches to said results cache. 
     
     
         7 . The apparatus of  claim 4  wherein said second and third searches are executed simultaneously. 
     
     
         8 . A computer apparatus comprising:
 a display;   a memory storing one or more computer programs;   an input device;   a data store storing configuration data and Unicode characters mapped from numerical values of performance metric data defining values over time of one or more attributes of one or more instances of one or more resource types in a system being monitored, said Unicode characters stored in one or more files of a non-relational database file system;   a programmable processor coupled to said memory and capable of being controlled by said computer program(s) stored therein, and coupled to said display, said input device, and said data store;   and wherein said one or more computer programs are structured to control said programmable processor to perform the following steps to implement a tree search of data in said data store:   D) receive a search query defining a relevant interval and first, second, third, fourth and fifth searches at two levels of nesting, said first level nesting specifed a nesting syntex block between said first and second searches, and said tree search signifying two separate traversals signified by a pair of tree search syntax blocks surrounding each of said second and third search and a pair of tree search syntax block surrounding said fourth and fifth search, said second and third search separated by a second nesting syntax block, and said fourth and fifth searches separated by a second nesting syntax block, said first search specifying a first resource type, peformance attribute and haying a filter criteria expressed in the form of a regular expression, said second search specifing a second resource type which is related to the first resource type, said third seach specifying a third resource type which is related to said second resource type, said fourth search specifying a fourth resource type which is related to said first resource type, said fifth search specifying a fifth resource type which is related to the fourth resource type, said second, third, fourth and fifth search also specifying a performance attribute, a filter condition specified as a regular expression; and   executing all said searches on said all levels of nesting and returning the results.   
     
     
         9 . The apparatus of  claim 8  wherein said computer programs are adapted to execute said by:
 passing the results of said first search to the second and fourth searches simultaneously; and 
 passing the results of said second search to said third search, and passing the results of said fourth search to said fifth search. 
 
     
     
         10 . A system comprising:
 a non relational database storing a plurality of time-series performance numbers that have been converted to text by conversion to Unicode such that any number of one or more digits has its value converted to its Unicode value, where there one time-series of Unicode values for each of the performance attributes of each instance of resource of a plurality of resources being monitored;   a processor coupled to said non relational database, programmed with a query to solicit from a user a start and stop time, and query request language that provides syntax blocks that include a syntax blocks allowing a user to specify a desired resource type, a performance attribute, a regular expression to receive filters and a nesting syntax block;   said processor programmed to receive a search query having a start and stop time, a first search consisting of a resource type, a performance attribute, a regular expression filter, a nesting syntax block, and a second search comprised of only a second attribute and a second regular expression filter,   said processor retrieving the necessary data and executing said searches and return the results; and   if either said regular expression contains a number or numerical expression, performing a reverse mapping from Unicode to numbers of any Unicode data retrieved to which said filter is to be applied before applying said filter.   
     
     
         11 . The system of  claim 10  wherein the step of retrieving the necessary data and reverse mapping comprises:
 reading the query to get the start and stop times; 
 reading a configuration file to determine what resources types exist or existed during said interval defined by said start and stop time and what resource types are related to other resources, and if a resource type existed only during part of relevant interval, determining from said configuration file when said resource type existed during the relevant interval; 
 creating a map of the resource types that match the resource type identified in said search query and which existed at any time during interval and related resource types; 
 loading the entire day(s) of performance attribute data for all the resource types for the relevant day(s) included in the interval; 
 parsing the query expression and determining what type of performance attribute data for the resource type named in the query on which the first search must be done; 
 determining the directory structure(s) that stores the performance attribute data from the date which contains the start and stop times, and determining the subdirectory of the relevant resource type, and determining the text files of the group of performance attributes data that contains the relevant performance attribute data, and determining from the configuration file what other resource types are related to the relevant resource and loading all this data into memory; 
 determining whether the relevant performance attribute data is stored in the cache and if so, moving relevant performance attributed data from the cache to the memory, and if not, sending a read request to the operating system API to read the entire day of the performance attribute data for the resource type including the text file for the group containing the relevant performance attribute data into memory; 
 to eliminate performance attribute data for attributes not named in the query and performance attribute data for the relevant attribute which is outside the relevant interval, building a new string of data containing only the relevant attribute data only during the relevant interval by finding the relevant row in the loaded file which contains the relevant performance attribute data and counting entries till the start time is reached and loading the performance attribute there into the new string and all subsequent attribute till the stop time is reached; and 
 applying the regular expression to the new string to find values which meet the criteria express in the regular expression for every relevant resource type, and if the regular expression contains a number or numerical range, decoding said Unicode values back to their original numeric values before applying said regular expression to them; 
 returning the relevant values and, if the have not already been decoded, decoding them from Unicode back to their original values, and if multiple values of multiple relevant resource types are found, then all the substrings are returned and decoded along with identifying information as to the resource type which returned each substring. 
 
     
     
         12 . A system comprising:
 a non relational database storing a plurality of time-series performance numbers that have been converted to text by conversion to Unicode such that any number of one or more digits has its value converted to its Unicode value, where there one time-series of Unicode values for each of the performance attributes of each instance of resource of a plurality of resources being monitored;   a processor coupled to said non relational database, programmed with a query to solicit from a user a start and stop time, and query request language that provides syntax blocks that include a syntax blocks allowing a user to specify a desired resource type, a performance or configuration attribute, a regular expression to receive filters and a nesting syntax block;   said processor programmed to receive a search query having a start and stop time, a first search consisting of a first resource type, a first performance or configuration attribute, a regular expression filter containing a first filter, a nesting syntax block, and a second search comprised of a second resource type which is related to the first resource type attribute, and a second performance or configuration attribute, and regular expression filter containing a second filter,   said processor retrieving the necessary data and executing said searches and return the results; and   if either said regular expression contains a number or numerical expression, performing a reverse mapping from Unicode to numbers of any Unicode data retrieved to which said filter is to be applied before applying said filter.