Systems and Methods for Reducing Data Storage Overhead by Utilizing Rolling Indices

Abstract

Systems and methods for reducing data storage overhead are disclosed herein. In some embodiments, a system includes a rollup service that converts a raw data set into a rolled up index that takes up less storage than the raw data but is created in such a way that the rolled up index can be queried so as to generate responses that will substantially correspond to responses that would be generated using the raw data.

What is claimed is: 
     
         1 . A method, comprising:
 obtaining raw data from one or more systems providing one or more services, the raw data comprising data elements having corresponding fields and metrics, the data elements being obtained according to a first time interval, the raw data having a storage size;   generating a rollup index of the raw data based on a first set of rollup parameters that comprises selections of the fields and metrics by:
 searching for matching ones of the data elements of the raw data that correspond to the first set of rollup parameters; 
 grouping the data elements together based on the matching; 
 flattening the data elements using a tree structure; and 
 rolling up the data elements into the rollup index using the tree structure based on a second time interval that is larger than the first time interval; 
   storing the rollup index in a storage space, the rollup index being reduced in size relative to the raw data so as to reduce an amount of the storage space required to store rollup index relative to storing the raw data; and   storing the raw data along with the rollup index, wherein the raw data comprises at least a last segment of the raw data covering a latter portion of the first time interval, wherein the rollup index comprises additional flattened data elements selected using a second set of rollup parameters, wherein the second set of rollup parameters are orthogonal in content to the first set of rollup parameters.   
     
     
         2 . The method according to  claim 1 , wherein when the first time interval is measured in milli-seconds or seconds, the second time interval is measured in any of days, weeks, months, years, or other time frames that are larger than seconds. 
     
     
         3 . The method according to  claim 1 , wherein the rollup index is stored alongside the raw data with both being available to search through a querying service. 
     
     
         4 . The method according to  claim 1 , wherein the selections of the fields and metrics guide a creation of at least one rollup job that is coalesced into a rollup index. 
     
     
         5 . The method according to  claim 1 , wherein the flattening of the data elements further comprises a creation of the aggregation trees and a grouping of data elements that are alike. 
     
     
         6 . The method according to  claim 4 , wherein the at least one rollup job runs continuously in the background. 
     
     
         7 . The method according to  claim 6 , further comprising:
 receiving a query having query parameters;   parsing the query parameters;   searching the query parameters against the rollup index; and   generating a response using the rollup index.   
     
     
         8 . The method according to  claim 7 , further comprising initially searching the last segment of the raw data for portions of the data elements that correspond to the query parameters prior to searching the rollup index. 
     
     
         9 . The method according to  claim 7 , wherein the response is generated relative to a third time interval that is larger than the second time interval. 
     
     
         10 . The method according to  claim 1 , wherein the rollup index is continually updated as the one or more systems generate new data elements. 
     
     
         11 . A system, comprising:
 a rollup service configured to:
 obtain raw data from one or more systems providing one or more services, the raw data comprising data elements having corresponding fields and metrics, the data elements being obtained according to a first time interval, the raw data having a storage size; 
 generate a rollup index of the raw data based on a first set of rollup parameters that comprises selections of the fields and metrics by:
 searching for matching ones of the data elements of the raw data that correspond to the first set of rollup parameters; 
 grouping the data elements together based on the matching; 
 flattening the data elements; and 
 rolling up the data elements for placement into the rollup index based on a second time interval that is larger than the first time interval; 
 
 store the rollup index in a storage space, the rollup index being reduced in size relative to the raw data so as to reduce an amount of the storage space required to store rollup index relative to storing the raw data; 
 store the raw data along with the rollup index, wherein the raw data comprises at least a last segment of the raw data covering a latter portion of the first time interval, wherein the rollup index comprises additional flattened data elements selected using a second set of rollup parameters, wherein the second set of rollup parameters are orthogonal in content to the first set of rollup parameters; and 
   a search service endpoint configured to:
 receive a query having query parameters; 
 parse the query parameters; 
 search the query parameters against the rollup index; and 
 generate a response using the rollup index. 
   
     
     
         12 . The system according to  claim 11 , wherein the search service endpoint configured to is further configured to initially search a last segment of the raw data for portions of the data elements that correspond to the query parameters prior to searching the rollup index. 
     
     
         13 . The system according to  claim 12 , wherein the response is generated relative to a third time interval that is larger than the second time interval. 
     
     
         14 . The system according to  claim 13 , wherein the rollup index is continually updated as the one or more systems generate new data elements. 
     
     
         15 . The system according to  claim 14 , wherein when the first time interval is measured in seconds the second time interval is measured in any of days, weeks, months, years, or other time frames that are larger than seconds. 
     
     
         16 . The system according to  claim 15 , wherein the rollup service is configured to store the raw data along with the rollup index. 
     
     
         17 . A method, comprising:
 obtaining raw historical data for a computing system, the raw historical data being obtained according to a first time interval;   converting the raw historical data into a rollup index, the rollup index comprising aggregations of data elements of the raw historical data which are grouped according to at least one field and at least one metric regarding the at least one field;   storing the raw historical data along with the rollup index, wherein the raw historical data comprises at least a last segment of the raw historical data covering a latter portion of the first time interval, wherein the rollup index comprises additional flattened data elements selected using a second set of rollup parameters, wherein the second set of rollup parameters are orthogonal in content to a first set of rollup parameters; and   generating a query response for a query using the rollup index, the query response substantially corresponding to a query response when executed against the raw historical data due to the conversion of the raw historical data into the rollup index.   
     
     
         18 . The method according to  claim 17 , wherein the raw historical data is converted into the rollup index using flattened aggregation trees. 
     
     
         19 . The method according to  claim 17 , wherein the aggregations of data elements in the rollup index are obtained relative to a second time interval that is greater than the first time interval, wherein the query response is generated relative to a third time interval that is greater than or equal to the second time interval. 
     
     
         20 . The method according to  claim 17 , wherein the rollup index is continually updated as one or more systems generate new data elements.