Purging of stored timeseries data

Abstract

There is disclosed methods, systems and computer program products for purging stored data in a repository. Users attach relative importance to all data samples across all timeseries in a repository. The importance attached to a data sample is the ‘utility value’ of the data sample. An algorithm uses the utility of data samples and allocates the storage space of the repository in such a way that the total loss of information due to purging is minimized while preserving samples with a high utility value.

1 . A method for purging timeseries data samples stored in a repository comprising:
 calculating a utility value for each said data sample;   determining information content of each said data sample; and   purging said stored data samples from said repository such that data samples having high utility value are retained and loss of information content of retained data samples is minimized.   
   
   
       2 . The method of  claim 1 , wherein said purging also ensures that a maximum capacity of said repository is not exceeded. 
   
   
       3 . The method of  claim 1 , wherein said utility value calculation is based on relationships between said stored data. 
   
   
       4 . The method of  claim 3 , wherein said utility value calculation is based on a dependency model being represented as a hierarchical graph having (i) nodes, and (ii) edges interconnecting nodes, said nodes and edges representing dependencies between said data samples, and each said data sample being associated with a node. 
   
   
       5 . The method of  claim 1 , wherein said utility value calculation is based on regions of said data samples of interest. 
   
   
       6 . The method of  claim 5 , wherein said utility value calculation is based on a dependency model of said data samples being represented as a hierarchical graph having (i) nodes, and (ii) edges interconnecting nodes, said nodes being associated with regions of interest. 
   
   
       7 . The method of  claim 1 , wherein said utility value calculation is based on the age of said data samples. 
   
   
       8 . The method of  claim 7 , wherein said utility value calculation is based on a dependency model of said data samples being represented as a hierarchical graph having (i) nodes, and (ii) edges interconnecting nodes, said nodes being associated with age. 
   
   
       9 . The method of  claim 8 , wherein age is determined by a polynomic function. 
   
   
       10 . The method of  claim 1 , wherein determination of said information content of said stored data samples is performed on the basis of using least one: of a probability distribution function, a mean square error, and a Kullback Liebler distance applied to said data samples. 
   
   
       11 . A method for purging stored timeseries data comprising:
 specifying meta-data purging policy rules and meta-data models of said stored data;   applying utility values to said stored data; and   purging said stored data samples from said repository such that data samples having high utility value are retained and loss of information content of retained data samples is minimized, and wherein said information content is determined based on said purging policy rules.   
   
   
       12 . A method for purging timeseries data samples stored in a repository comprising:
 calculating a utility value for each said data samples;   determining information content of each said data sample; and   purging said stored data samples from said repository such that data samples having high utility value are retained and loss of information content of retained data samples is minimized, and such that a maximum capacity of said repository is not exceeded.   
   
   
       13 . The method of  claim 12 , wherein said utility value calculation is based on relationships between said stored data. 
   
   
       14 . The method of  claim 13 , wherein said utility value calculation is based on a dependency model being represented as a hierarchical graph having (i) nodes, and (ii) edges interconnecting nodes, said nodes and edges representing dependencies between said data samples, and each said data sample being associated with a node. 
   
   
       15 . The method of  claim 12 , wherein said utility value calculation is based on regions of said timeseries data samples of interest. 
   
   
       16 . The method of  claim 15 , wherein said utility value calculation is based on a dependency model of said data samples being represented as a hierarchical graph having (i) nodes, and (ii) edges interconnecting nodes, said nodes being associated with regions of interest. 
   
   
       17 . The method of  claim 12 , wherein said utility value calculation is based on the age of said data samples. 
   
   
       18 . The method of  claim 17 , wherein said utility value calculation is based on a dependency model of said data samples being represented as a hierarchical graph having (i) nodes, and (ii) edges interconnecting nodes, said nodes being associated with age. 
   
   
       19 . The method of  claim 12 , wherein determination of said information content of said stored data samples is performed on the basis of using least one: of a probability distribution function, a mean square error, and a Kullback Liebler distance applied to said data samples. 
   
   
       20 . A system comprising:
 a repository storing timeseries data samples; and   a processor for calculating a utility value for each said data samples, determining information content of each said data sample, and purging said stored data samples from said repository such that data samples having high utility value are retained and loss of information content of retained data samples is minimized.   
   
   
       21 . The system of  claim 20 , wherein said purging also ensures that a maximum capacity of said repository is not exceeded. 
   
   
       22 . The system of  claim 20 , wherein said utility value calculation is based on relationships between said stored data. 
   
   
       23 . The system of  claim 22 , wherein said utility value calculation is based on a dependency model being represented as a hierarchical graph having (i) nodes, and (ii) edges interconnecting nodes, said nodes and edges representing dependencies between said data samples, and each said data samples being associated with a node. 
   
   
       24 . The system of  claim 20 , wherein said utility value calculation is based on regions of said data samples of interest. 
   
   
       25 . The system of  claim 24 , wherein said utility value calculation is based on a dependency model of said data samples being represented as a hierarchical graph having (i) nodes, and (ii) edges interconnecting nodes, said nodes being associated with regions of interest. 
   
   
       26 . The system of  claim 20 , wherein said utility value calculation is based on the age of said data samples. 
   
   
       27 . The system of  claim 26 , wherein said utility value calculation is based on a dependency model of said data samples being represented as a hierarchical graph having (i) nodes, and (ii) edges interconnecting nodes, said nodes being associated with age. 
   
   
       28 . The system of  claim 20 , wherein determination of said information content of said stored data samples is performed on the basis of using least one: of a probability distribution function, a mean square error, and a Kullback Liebler distance applied to said data samples. 
   
   
       29 . A system for purging stored data comprising:
 a repository storing timeseries data samples;   a memory specifying meta-data purging policy rules and meta-data models of said stored data samples; and   a processor applying utility values to said stored data, and purging said stored data samples from said repository such that data samples having high utility value are retained and loss of information content of retained data samples is minimized, and wherein said information content is determined based on said purging policy rules.   
   
   
       30 . A system for purging stored data, comprising:
 a repository storing timeseries data samples; and   a processor calculating a utility value for each said data samples, determining information content of each said data sample, and purging said stored data samples from said repository such that data samples having high utility value are retained and loss of information content of retained data samples is minimized, and such that a maximum capacity of said repository is not exceeded.   
   
   
       31 . A computer program product comprising a computer useable medium having a computer readable program, wherein the computer readable program when executed on a computer causes the computer to:
 calculate a utility value for each said data samples;   determine information content of each said data sample; and   purge said stored data samples from said repository such that data samples having high utility value are retained and loss of information content of retained data samples is minimized and such that a maximum capacity of said repository is not exceeded.