Data ingestion stager for time series database

Abstract

A time series data stager that receives input data sets and outputs output data blocks for ingestion into a time series database, with the out data blocks being sent at timings according to a sliding window based on a predetermined time.

What is claimed is: 
     
       1. A method comprising:
 receiving, from a set of client data driver(s) and by a time series stager system, a plurality of input data sets designed for ingestion into a time series database, with each input data set including payload data and a timestamp related to collection of the payload data; 
 for each given input data set of the plurality of input data sets, storing the given input data set a data storage device in the time series stager system for at least a predetermined interval of time; 
 for each given input data set of the plurality of input data sets, outputting the given input data set to a time series database system in an order, relative to other input data sets of the plurality of input data sets, determined by respective timestamps of the plurality of input data sets; 
 for each given input data set of the plurality of input data sets, responsive to outputting of the given input data set, ceasing to store the given input data set in the data storage device of the time series stager system; 
 receiving, from a client query driver, by the time series stager system and at a first point in time, a first query; 
 performing the first query against input data sets stored in the data storage device of the time series stager system at the first point in time to yield first query results; and 
 outputting, by the time series stager and to the client query driver, the first query results. 
 
     
     
       2. The method of  claim 1  wherein:
 the receipt of the plurality of input data sets occurs through an ingestion thread operating in a processor(s) set of the time series stager system. 
 
     
     
       3. The method of  claim 1 
 the outputting of the input data sets occurs through a shipping thread operating in a processor(s) set of the time series stager system. 
 
     
     
       4. The method of  claim 1  wherein the outputting of the first query results occurs through a query thread operating in a processor(s) set of the time series stager system. 
     
     
       5. The method of  claim 1  further comprising:
 maintaining an index of current input data sets of the plurality of input data sets that are currently stored in the data storage device of the time series stager system; 
 wherein the performance of the first query includes checking the index to determine the first query results. 
 
     
     
       6. The method of  claim 1  wherein:
 the data storage device includes a non-volatile data storage medium; and 
 the storage of the plurality of input data sets is non-volatile type data storage. 
 
     
     
       7. A computer program product (CPP) comprising:
 a computer readable storage medium; and 
 computer code stored on the computer readable storage medium, the computer code including instructions and data for causing a processor(s) set to perform at least the following operations:
 receiving, from a set of client data driver(s) and by a time series stager system, a plurality of input data sets designed for ingestion into a time series database, with each input data set including payload data and a timestamp related to collection of the payload data, 
 for each given input data set of the plurality of input data sets, storing the given input data set a data storage device in the time series stager system for at least a predetermined interval of time, 
 for each given input data set of the plurality of input data sets, outputting the given input data set to a time series database system in an order, relative to other input data sets of the plurality of input data sets, determined by respective timestamps of the plurality of input data sets, 
 for each given input data set of the plurality of input data sets, responsive to outputting of the given input data set, ceasing to store the given input data set in the data storage device of the time series stager system, 
 receiving, from a client query driver, by the time series stager system and at a first point in time, a first query, 
 performing the first query against input data sets stored in the data storage device of the time series stager system at the first point in time to yield first query results, and 
 outputting, by the time series stager and to the client query driver, the first query results. 
 
 
     
     
       8. The CPP of  claim 7  wherein the receipt of the plurality of input data sets occurs through an ingestion thread operating in a processor(s) set of the time series stager system. 
     
     
       9. The CPP of  claim 7  wherein the outputting of the input data sets occurs through a shipping thread operating in a processor(s) set of the time series stager system. 
     
     
       10. The CPP of  claim 7  wherein the outputting of the first query results occurs through a query thread operating in a processor(s) set of the time series stager system. 
     
     
       11. The CPP of  claim 7  further comprising:
 maintaining an index of current input data sets of the plurality of input data sets that are currently stored in the data storage device of the time series stager system; 
 wherein the performance of the first query includes checking the index to determine the first query results. 
 
     
     
       12. The CPP of  claim 7  wherein:
 the data storage device includes a non-volatile data storage medium; and 
 the storage of the plurality of input data sets is non-volatile type data storage. 
 
     
     
       13. A computer system (CS) comprising:
 a processor(s) set; 
 a computer readable storage medium; and 
 computer code stored on the computer readable storage medium, the computer code including instructions and data for causing the processor(s) set to perform at least the following operations:
 receiving, from a set of client data driver(s) and by a time series stager system, a plurality of input data sets designed for ingestion into a time series database, with each input data set including payload data and a timestamp related to collection of the payload data, 
 for each given input data set of the plurality of input data sets, storing the given input data set a data storage device in the time series stager system for at least a predetermined interval of time, 
 for each given input data set of the plurality of input data sets, outputting the given input data set to a time series database system in an order, relative to other input data sets of the plurality of input data sets, determined by respective timestamps of the plurality of input data sets, 
 for each given input data set of the plurality of input data sets, responsive to outputting of the given input data set, ceasing to store the given input data set in the data storage device of the time series stager system, 
 receiving, from a client query driver, by the time series stager system and at a first point in time, a first query, 
 performing the first query against input data sets stored in the data storage device of the time series stager system at the first point in time to yield first query results, and 
 outputting, by the time series stager and to the client query driver, the first query results. 
 
 
     
     
       14. The CS of  claim 13  wherein the receipt of the plurality of input data sets occurs through an ingestion thread operating in a processor(s) set of the time series stager system. 
     
     
       15. The CS of  claim 13  wherein the outputting of the input data sets occurs through a shipping thread operating in a processor(s) set of the time series stager system. 
     
     
       16. The CS of  claim 13  wherein the outputting of the first query results occurs through a query thread operating in a processor(s) set of the time series stager system. 
     
     
       17. The CS of  claim 13  further comprising:
 maintaining an index of current input data sets of the plurality of input data sets that are currently stored in the data storage device of the time series stager system; 
 wherein the performance of the first query includes checking the index to determine the first query results. 
 
     
     
       18. The CS of  claim 13  wherein:
 the data storage device includes a non-volatile data storage medium; and 
 the storage of the plurality of input data sets is non-volatile type data storage.