Systems and methods for storing and retrieving data sets based on temporal information

Abstract

Described herein are systems and methods for providing data sets from a constantly changing database to a streaming machine learning component. In one embodiment, a data streaming sub-system receives multiple incoming streams of data sets, in which each stream is generated in real-time by one of multiple data sources. The streaming sub-system sends data sets, on-the-fly as they are received, to storage in the memory of a database, in which there is a linkage between the storage and the time of arrival or the time of storage, of the data sets. The database receives, from a machine learning component, a request to receive data sets according to a particular time or time period. In response to such request, the database identifies such data sets according to the particular time or time period and sends them to the machine learning component.

What is claimed is: 
     
         1 . A method for efficiently providing data sets to a streaming machine learning component from a constantly changing database, comprising:
 receiving, in a data streaming sub-system, a plurality of incoming streams of data sets, in which each of the streams is generated in real time by one of a plurality of data sources to be analyzed in real-time using a streaming machine learning process;   sending the data sets, by the data streaming sub-system, on-the-fly as the data sets are received, to storage in a memory belonging to a database and associated with the streaming sub-system, in which said storage is affected by a time of arrival or time of storage associated with each of the data sets, such that a linkage is created between each of the data sets received and stored and said time of arrival or time of storage associated with that particular data set;   receiving, in the database, from a machine learning component, a request to receive those of the data sets received and stored in the database during or in a particular time or time period;   identifying, by the database, using said linkage, said those of the data sets received and stored in the database during or in the particular time or time period; and   sending, by the database, to the machine learning component, said data sets identified, thereby facilitating said analysis of the data sources in real time.   
     
     
         2 . The method of  claim 1 , wherein said linkage is associated with a function that maps each of the data sets to a particular storage location within the memory as a function of the respective time of arrival or time of storage, in which said sending of the data sets to storage is done according to said function and in conjunction with said particular storage location. 
     
     
         3 . The method of  claim 2 , wherein said function is a hash function, in which said time of arrival or time of storage is used to calculate, via the hash function, the particular storage location of the respective data set. 
     
     
         4 . The method of  claim 3 , wherein:
 said data base is an in-memory data base;   said memory comprises a plurality of memory modules, in which each of the memory modules is a random access memory; and   said hash function is used in conjunction with said time of arrival or time of storage to determine both the particular memory module to store the respective data set and the particular address within the particular memory module to store said respective data set.   
     
     
         5 . The method of  claim 4 , wherein said hash function is a single hash function operative to determine both the particular memory module to store the respective data set and the particular address within the particular memory module to store said respective data set. 
     
     
         6 . The method of  claim 4 , wherein said hash function comprises a first hash function and a second hash function, in which the first hash function is operative to determine the particular memory module to store the respective data set, and the second hash function is operative to determine the particular address within the particular memory module to store said respective data set, in which both the first and second hash functions operate on the same time of arrival or time of storage of the respective data set. 
     
     
         7 . The method of  claim 4 , wherein said hash function is operative to both (i) facilitate said linkage between each of the data sets and said time of arrival or time of storage associated with that particular data set and (ii) facilitate a load-balanced distribution of said data sets among said plurality of memory modules. 
     
     
         8 . The method of  claim 2 , wherein said function is a tabular mapping between said time of arrival or time of storage and the particular storage location of the respective data set. 
     
     
         9 . The method of  claim 2 , wherein both said sending and said identification are done using the same function. 
     
     
         10 . The method of  claim 9 , wherein said functions is available to both the data streaming sub-system and the database, thereby enabling said identification. 
     
     
         11 . The method of  claim 10 , wherein said data streaming sub-system is separate from the database. 
     
     
         12 . The method of  claim 10 , wherein said data streaming sub-system is a part of the database. 
     
     
         13 . A system operative to efficiently provide data sets to a streaming machine learning component from a constantly changing database, comprising:
 a database comprising a memory and a data-set-identifier; and   a data streaming sub-system comprising a mapping-element, in which the data streaming subsystem is operative to: (i) receive a plurality of incoming streams of data sets, in which each of the streams is generated in real time by one of a plurality of data sources to be analyzed in real-time using a streaming machine learning process and (ii) send the data sets, on-the-fly as the data sets are received, to storage in the memory, in which said storage and respective storage location within the memory are directed by a time of arrival or time of storage associated with each of the data sets, in which said direction is facilitated by the mapping-element executing a mapping function, such that a linkage is created between the storage location of each of the data sets stored and said time of arrival or time of storage associated with that particular data set;   wherein the database is configured to:   get, from a machine learning component, a request to receive those of the data sets received and stored in the database during or in a particular time or time period;   identify, using said linkage and via the data-set-identifier executing the same mapping function, said those of the data sets received and stored in the database during or in the particular time or time period; and   send, to the machine learning component, said data sets identified, thereby facilitating said analysis of the data sources in real time.   
     
     
         14 . A system operative to efficiently provide data sets to a streaming machine learning component from a constantly changing database, comprising:
 a plurality of incoming streams of data sets, in which each of the streams is generated in real time by one of a plurality of data sources to be analyzed in real-time using a streaming machine learning process;   a memory operative to act as a medium for storing the data sets and serving the data sets in real-time to the streaming machine learning process;   a data streaming sub-system configured to receive from the plurality of data sources the plurality of data sets, in real time, as the incoming streams of data, and to distribute the data sets for storage in the memory on-the-fly as the data sets are received; and   a smart database comprising said memory and a real-time tracking mechanism configured to: (i) continuously track said reception and distribution of the data sets and (ii) based on said tracking to continuously generate and update a temporal metadata associated with the plurality of incoming streams of data sets, in which the temporal metadata is operative to distinguish between newer data sets and older data sets in the plurality of data sets currently stored in the memory;   wherein:   the system is associated with a streaming machine learning component configured to interact with at least some of the plurality of data sets in the memory, thereby executing said streaming machine learning process in conjunction with at least some of the data sets and thus facilitating said analysis of the data sources in real time;   the streaming machine learning component is further configured to request, during said interaction, from the smart database, a certain amount or a certain volume of the most recent ones of the data sets currently in the memory; and   the smart database, as a response to said request, is configured to: (i) use the temporal metadata to identify in the memory said certain amount or certain volume of the most recent ones of the data sets, and then (ii) send the data sets identified to the streaming machine learning component.   
     
     
         15 . A smart database system operative to continuously generate and update temporal metadata associated with incoming streams of data sets, comprising:
 a memory operative to act as a medium for storing streams of data sets received in the system; and   a real-time tracking mechanism configured to: (i) continuously track reception and distribution of the data sets in conjunction with the memory and (ii) based on said tracking to continuously generate and update a temporal metadata associated with the streams of data sets received, in which the temporal metadata is operative to distinguish between newer data sets and older data sets in the data sets currently stored in the memory;   wherein:   the smart database system, as a response to a request for a certain amount or a certain volume of the most recent ones of the data sets currently in the memory, is configured to: (i) use the temporal metadata to identify in the memory said certain amount or certain volume of the most recent ones of the data sets, and then (ii) send the data sets identified.   
     
     
         16 . A method for efficiently providing data sets to a streaming machine learning component from a constantly changing database, comprising:
 continuously tracking, by a real-time tracking mechanism belonging to a database, a process of continuously adding a plurality of data sets into the database, in which the plurality of data sets are streamed into the database from a plurality data sources to be analyzed in real time by a streaming machine learning component;   continuously generating and updating, by the real-time tracking mechanism, based on said continuous tracking, a temporal metadata associated with the plurality of data sets, in which the temporal metadata is operative to associate a given time or a time period with those of the data sets received and stored in the database during or in said time or time period;   receiving, in the database, from a machine learning component, a request to receive those of the data sets received and stored in the database during or in a particular time or time period;   identifying, by the database, using the temporal metadata, said those of the data sets received and stored in the database during or in the particular time or time period; and   sending, by the database, to the machine learning component, said data sets identified, thereby facilitating said analysis of the data sources in real time.   
     
     
         17 . The method of  claim 16 , wherein said continuously tracking comprises continuously tracking storage locations of the data sets within the database. 
     
     
         18 . The method of  claim 17 , wherein said continuously tracking storage locations of the data sets within the database comprises continuously tracking a specific storage location of each of the data sets within the database. 
     
     
         19 . The method of  claim 17 , wherein said continuously tracking a storage location of the data sets within the database comprises continuously tracking a general storage location of each of the data sets within the database, in which said general storage location comprises at least one of: a certain address span, a certain sector span, and a particular storage element within the database. 
     
     
         20 . The method of  claim 17 , wherein said temporal metadata comprises a table associating at least one particular time or time period with the storage locations of those of the data sets received and stored in the database during or in said particular time or time period.