System for organizing and fast searching of massive amounts of data

Abstract

A system to collect and store in a special data structure arranged for rapid searching massive amounts of data. Performance metric data is one example. The performance metric data is recorded in time-series measurements, converted into unicode, and arranged into a special data structure having one directory for every day which stores all the metric data collected that day. The performance metric data is collected by one or more probes running on machines about which data is being collected. The performance metric data is compressed prior to transmission to a server over any data path. The data structure at the server where analysis is done has a subdirectory for every resource type. Each subdirectory contains text files of performance metric data values measured for attributes in a group of attributes to which said text file is dedicated. Each attribute has its own section and the performance metric data values are recorded in time series as unicode hex numbers as a comma delimited list. Analysis of the performance metric data is done using regular expressions. For speed, cache memory is used. Performance metric data outside the start time and end time named by the user on a query screen is eliminated before the regular expression is applied.

What is claimed is: 
     
         1 . An apparatus comprising:
 a memory storing performance metric data of one of more resource types, each resource type having one or more attributes, said performance metric data having been converted to Unicode prior to storage;   a processor programmed with at least an operating system and a search program for searching said memory;   said search program having the capability to receive one or more partial syntaxes, each comprised of a resource type, an attribute and a regular expression giving search conditions; and   said search program capable of receiving two or more user-defined partial syntax expressions, one or more of said partial syntax expression for resource types which are related to the resource type of a first partial syntax, and capable of searching the first partial syntax for results which qualify, and applying said results as the starting point for evaluating said one or more related resource type partial syntaxes for results which qualify, and merging the evaluation results of all the partial syntaxes.   
     
     
         2 . [user defined start time and stop time] The apparatus of  claim 2  wherein said search program is capable of reading one or more time range boxes which contain a user supplied start time and stop time for said search and restricting data accessed from said memory to data gathered between and including said start time and said stop time. 
     
     
         3 . [reverse mapping] The apparatus of  claim 2  wherein said search program is capable of recognizing performance metric data in the form of numbers in said regular expression expressing filter conditions in each partial syntax and converting each said performance metric number to Unicode before carrying out said search. 
     
     
         4 . [template search] The apparatus of  claim 1  where said search program is capable of recognizing identification codes in said regular expression representing prearranged searches which are stored in advance and is capable of generating the proper syntax for an identification code to cause said search program to implement the appropriate prearranged search. 
     
     
         5 . [user defined variables in template search] The apparatus of  claim 4  wherein said search program is capable of recognizing user defined variables for one or more variables in a template search identified by an identification code and substituting said user defined variable(s) into said template search before running said search. 
     
     
         6 . [plain language search] The apparatus of  claim 1  wherein said search program is capable of accepting plain language searches and converting them to the proper syntax for the partial syntaxes before executing said search. 
     
     
         7 . [data in said specially defined file system] The apparatus of  claim 1  wherein said memory stores said Unicode in a file system organized with one top level directory per unit of time where a unit of time is the time over which performance metric data is gathered for one attribute, one performance metric number gathered per one subunit of time, said top level directory storing one or more subdirectories, one per resource type, each subdirectory storing one or more files of Unicode data gathered from said one or more attributes of said resource type to which said subdirectory is devoted, each said file having a one or more sections, each for storing the data from one of the related attributes of said resource in a plurality of time slots, each for storing one Unicode letter and each of which corresponds to the timeslot during which the underlying performance number was gathered. 
     
     
         8 . [metadata for each performance number stored in the file system itself] The apparatus of  claim 7  wherein each item of performance metric data has metadata comprising at least the time of day during which it was gathered, the attribute from which it was gathered, the date on which it was gathered, and wherein the metadata of each performance metric number is stored in the file system itself. 
     
     
         9 . An apparatus comprising:
 a memory storing performance metric data of one of more resource types, each resource type having one or more attributes, said performance metric data having been converted to Unicode prior to storage;   a processor programmed with at least an operating system and a search program for searching said memory;   said search program having the capability to receive one or more partial syntaxes, each comprised of a resource type, an attribute and a regular expression giving search conditions; and   said search program capable of receiving first, second, third, fourth and fifth or more user-defined partial syntax expressions, said second and fourth partial syntax expression for resource types which are related to the resource type of said first partial syntax, and third and fifth partial syntaxes being for resource types which are related to said second and fourth partial syntax, respectively, said search program being capable of searching the first partial syntax for results which qualify, and applying said results as the starting point for evaluating whether said second and fourth partial syntaxes have results which qualify, said search program being capable of using the results which qualify by using said second and fourth partial syntaxes and applying said results as the starting point for evaluating whether said third and fifth partial syntaxes have results which qualify, and merging the qualifying results of all the partial syntaxes.   
     
     
         10 . [user defined start time and stop time] The apparatus of  claim 9  wherein said search program is capable of reading one or more time range boxes which contain a user supplied start time and stop time for said search and restricting data accessed from said memory to data gathered between and including said start time and said stop time. 
     
     
         11 . [reverse mapping] The apparatus of  claim 9  wherein said search program is capable of recognizing performance metric data in the form of numbers in said regular expression expressing filter conditions in each partial syntax and converting each said performance metric number to Unicode before carrying out said search. 
     
     
         12 . [template search] The apparatus of  claim 9  where said search program is capable of recognizing identification codes in said regular expression representing prearranged searches which are stored in advance and is capable of generating the proper syntax for an identification code to cause said search program to implement the appropriate prearranged search. 
     
     
         13 . [user defined variables in template search] The apparatus of  claim 12  wherein said search program is capable of recognizing user defined variables for one or more variables in a template search identified by an identification code and substituting said user defined variable(s) into said template search before running said search. 
     
     
         14 . [plain language search] The apparatus of  claim 9  wherein said search program is capable of accepting plain language searches and converting them to the proper syntax for the partial syntaxes before executing said search. 
     
     
         15 . [data in said specially defined file system] The apparatus of  claim 9  wherein said memory stores said Unicode in a file system organized with one top level directory per unit of time where a unit of time is the time over which performance metric data is gathered for one attribute, one performance metric number gathered per one subunit of time, said top level directory storing one or more subdirectories, one per resource type, each subdirectory storing one or more files of Unicode data gathered from said one or more attributes of said resource type to which said subdirectory is devoted, each said file having a one or more sections, each for storing the data from one of the related attributes of said resource in a plurality of time slots, each for storing one Unicode letter and each of which corresponds to the timeslot during which the underlying performance number was gathered. 
     
     
         16 . [metadata for each performance number stored in the file system itself] The apparatus of  claim 15  wherein each item of performance metric data has metadata comprising at least the time of day during which it was gathered, the attribute from which it was gathered, the date on which it was gathered, and wherein the metadata of each performance metric number is stored in the file system itself. 
     
     
         17 . An apparatus comprising:
 a memory storing performance metric data of one of more resource types, each resource type having one or more attributes, said performance metric data having been converted to Unicode prior to storage;   a processor programmed with at least an operating system and a search program for searching said memory;   said search program having the capability to receive one or more partial syntaxes, each comprised of a resource type, an attribute and a regular expression giving search conditions; and   said search program capable of receiving first, second, third, fourth, fifth, sixth and seventh or more user-defined partial syntax expressions,
 said second partial syntax expression for a resource type which is related to the resource type of said first partial syntax, 
 and third partial syntax being for a resource type which is related to said resource type of second partial syntax, 
 said fourth partial syntax expression being for a resource type which is related to the resource type of said third partial syntax, 
 said fifth partial syntax expression being for a resource which is related to the resource type of said second partial syntax, 
 said sixth partial syntax expression being for a resource type which is related to the resource type of said first partial syntax, 
 said seventh partial syntax expression being for a resource type which is related to the resource type of said sixth partial syntax, 
   said search program being capable of searching the first partial syntax for results which qualify, and applying said results as the starting point for evaluating whether said second and sixth partial syntaxes have results which qualify,   said search program being capable of using the results which qualify by using said sixth partial syntaxes and applying said results as the starting point for evaluating whether said seventh partial syntax have results which qualify,   said search program being capable of using the results which qualify when using said second partial syntax as the starting point for evaluating whether said third and fifth partial syntaxes have results which qualify,   said search program being capable of using the results which qualify when using said third partial syntax as the starting point for evaluating whether said fourth partial syntax has results which qualify,   and merging the qualifying results of evaluation of said:
 first, second, third and fourth partial syntaxes, and 
 first, second and fifth partial syntaxes, and 
 first, sixth and seventh partial syntaxes. 
   
     
     
         18 . [user defined start time and stop time] The apparatus of  claim 17  wherein said search program is capable of reading one or more time range boxes which contain a user supplied start time and stop time for said search and restricting data accessed from said memory to data gathered between and including said start time and said stop time. 
     
     
         19 . [reverse mapping] The apparatus of  claim 17  wherein said search program is capable of recognizing performance metric data in the form of numbers in said regular expression expressing filter conditions in each partial syntax and converting each said performance metric number to Unicode before carrying out said search. 
     
     
         20 . [template search] The apparatus of  claim 17  where said search program is capable of recognizing identification codes in said regular expression representing prearranged searches which are stored in advance and is capable of generating the proper syntax for an identification code to cause said search program to implement the appropriate prearranged search. 
     
     
         21 . [user defined variables in template search] The apparatus of  claim 17  wherein said search program is capable of recognizing user defined variables for one or more variables in a template search identified by an identification code and substituting said user defined variable(s) into said template search before running said search. 
     
     
         22 . [data in said specially defined file system] The apparatus of  claim 17  wherein said memory stores said Unicode in a file system organized with one top level directory per unit of time where a unit of time is the time over which performance metric data is gathered for one attribute, one performance metric number gathered per one subunit of time, said top level directory storing one or more subdirectories, one per resource type, each subdirectory storing one or more files of Unicode data gathered from said one or more attributes of said resource type to which said subdirectory is devoted, each said file having a one or more sections, each for storing the data from one of the related attributes of said resource in a plurality of time slots, each for storing one Unicode letter and each of which corresponds to the timeslot during which the underlying performance number was gathered. 
     
     
         23 . [metadata for each performance number stored in the file system itself] The apparatus of  claim 22  wherein each item of performance metric data has metadata comprising at least the time of day during which it was gathered, the attribute from which it was gathered, the date on which it was gathered, and wherein the metadata of each performance metric number is stored in the file system itself.