Techniques for ingesting time-series based event data

Abstract

Aspects of the present disclosure relate to ingesting event data into a discoverable event stream using a common schema to assign timeseries properties to the event data. These techniques may include receiving a first event of a pre-processed event stream, the first event including a plurality of event data fields representing an occurrence of the first event, and determining timing information based on the plurality of event data fields representing the occurrence of the first event. In addition, the techniques may include generating, based on combining the plurality of event data fields and the timing information, a second event of a discoverable event stream corresponding to the pre-processed event stream, and generating, within a time series of the discoverable event stream, based on the timing information and an event field of the plurality of event data fields, a time series entry corresponding to the second event.

What is claimed is: 
     
         1 . A method comprising:
 receiving a first event of a pre-processed event stream, the first event including a plurality of event data fields representing an occurrence of the first event;   determining timing information based on the plurality of event data fields representing the occurrence of the first event;   generating, based on combining the plurality of event data fields and the timing information, a second event of a discoverable event stream corresponding to the pre-processed event stream; and   generating, within a time series of the discoverable event stream, based on the timing information and an event field of the plurality of event data fields, a time series entry corresponding to the second event.   
     
     
         2 . The method of  claim 1 , wherein determining the timing information comprises:
 determining the timing information based on a plurality of events received over the pre-processed event stream.   
     
     
         3 . The method of  claim 1 , wherein determining the timing information comprises:
 identifying a timestamp generated by a client application within the first event;   determining a correction value based upon a local system clock and the timestamp; and   determining the timing information based upon the correction value.   
     
     
         4 . The method of  claim 1 , wherein determining the timing information comprises:
 identifying a highest timestamp amongst a plurality of events of the pre-processed event stream;   determining a correction value based upon a local system clock and highest timestamp; and   determining the timing information based upon the correction value.   
     
     
         5 . The method of  claim 4 , wherein determining the correction value comprises subtracting the highest timestamp from the local system clock to determine the correction value. 
     
     
         6 . The method of  claim 1 , wherein generating the second event of the discoverable event stream corresponding to the pre-processed event stream, comprises:
 generating one or more common data fields based at least in part on the plurality of event data fields, the one or more common data fields shared across a plurality of discoverable event streams having different data sources; and   generating, based on combining the plurality of event data fields and the one or more common data fields, the second event of the discoverable event stream corresponding to the pre-processed event stream.   
     
     
         7 . The method of  claim 6 , wherein the one or more common data fields include an event identifier identifying an event, a name of the event, a first timestamp indicating the time of occurrence of the event, a source identifier identifying a source of the event, a second timestamp indicating an arrival time of the event data corresponding to the event at an ingestion component, and/or a tenant identifier identifying a tenant of a multi-tenant system associated with the event. 
     
     
         8 . The method of  claim 1 , further comprising:
 identifying a namespace associated with an activation trigger;   identifying a plurality of time-series based objects of the discoverable event stream associated with the namespace;   determining that an activation rule is met over the plurality of time-series based objects;   transmitting an activation trigger event to an activation component, the activation trigger event including a generic activation identifier identifying an action to perform, and an actor identifier identifying an actor to perform the action;   receiving an activation trigger event via a discoverable event stream, the activation trigger event including a generic activation identifier and an actor identifier;   generating, based on the generic activation identifier and the actor identifier, an activation instruction event for causing an action associated with the generic activation identifier at an actor corresponding to the actor identifier, wherein activation instruction event is a first activation instruction event, the action is a first action;   configuring, based on the activation trigger event, the activation instruction event to trigger the action at the actor;   transmitting the activation instruction event to the actor based on the actor identifier;   generating a second activation instruction event for causing a second action associated with the actor;   identifying a conflict between the first action and the second action; and   determining, based on the conflict, not to send the second activation instruction event to the actor.   
     
     
         9 . The method of  claim 1 , further comprising:
 determining historical query information based upon historical execution of a query over the discoverable event stream;   identifying a storage attribute of data associated with the historical execution of the query over the discoverable event stream, the storage attribute indicating at least one of a size of the data or an assigned storage location of the data in a first storage location or a second storage location;   identifying an execution attribute of a future execution the query over the discoverable event stream; and   modifying the assigned storage location of the data based on the historical query information, the storage attribute, and the execution attribute.   
     
     
         10 . The method of  claim 1 , further comprising:
 executing, by a processor, a query for event data in the discoverable event stream, wherein the event data is stored as timeseries data in the discoverable event stream such that a property corresponding to an object has multiple different values at multiple different time instances;   determining, based on a timeseries function, a subset of the multiple different values for the property in instances of the event data that correspond to the multiple different time instances to return for the query; and   returning the subset of the multiple different values for the property for the query.   
     
     
         11 . The method of  claim 10 , further comprising:
 determining, from the query, subqueries that result in multiple intermediate states of collecting the instances of event data;   separately executing each of the subqueries and storing results of the corresponding intermediate state of the multiple intermediate states, wherein executing each of the subqueries is spread over time to provide controlled resource impact; and   combining the results as part of executing the query to identify the instances of event data.   
     
     
         12 . The method of  claim 10 , further comprising:
 creating, by the processor based on the instances of event data identified from the discoverable event stream, a new event for adding to the discoverable event stream;   adding, by the processor, the new event to the discoverable event stream including applying a timestamp to the new event in the discoverable event stream, wherein the query indicates a comparison for a value of the property, and wherein adding the new event is based at least in part on an outcome of the comparison for the value of the property for at least a portion of the event data.   
     
     
         13 . The method of  claim 1 , further comprising:
 generating, based on an activation trigger event, by an activation component, a first activation instruction event to trigger a first action at a first actor;   transmitting the first activation instruction event to the first actor;   monitoring for receipt of an actor status event of the discoverable event stream, the actor status event indicating performance or non-performance of the first action by the first actor in response to the first activation instruction event;   generating, based on the monitoring, by the activation component, a second activation instruction event to trigger a second action at a second actor; and   transmitting the second activation instruction event to the second actor.   
     
     
         14 . The method of  claim 1 , further comprising:
 obtaining, by a processor based on a query over a time window of event data in the discoverable event stream, instances of the event data related to an object and based on a value of a property indicated in the instances of the event data; and   displaying, via a user interface, multiple stacked visual representations related to the instances of the event data, wherein at least one of the multiple stacked visual representations of the event data is derived from at least another one of the multiple stacked visual representations of the event data using a time-series function.   
     
     
         15 . The method of  claim 1 , further comprising:
 receiving a first event of the discoverable event stream, the first event including a plurality of event data fields representing an occurrence of the first event;   identifying a discoverable event stream view associated with an event data field of the plurality of event data fields; and   generating, based on application of a modification to or computation on the event data field, a second event of the discoverable event stream view.   
     
     
         16 . The method of  claim 1 , further comprising:
 executing, by a processor, a query for event data in the discoverable event stream over a time window that includes a delay tolerance window specific to the query, wherein the event data is stored as timeseries data in the discoverable event stream such that at least one property corresponding to at least one object has multiple different values at multiple different time instances, and wherein the delay tolerance window is configured to capture potentially out-of-order event data or late arriving event data; and   returning a subset of multiple different values for the at least one property in instances of the event data corresponding to the time window with the delay tolerance window applied for the query,   wherein executing the query on the event data comprises executing the query as the event data arrives in the discoverable event stream based on a first value of an execution efficiency metric,   wherein the execution efficiency metric is based on one or more of a type of the query, operators in the query, or a number of results for the query.   
     
     
         17 . An apparatus, comprising:
 a memory; and   a processor coupled with the memory and configured to:
 receive a first event of a pre-processed event stream, the first event including a plurality of event data fields representing an occurrence of the first event; 
 determine timing information based on the plurality of event data fields representing the occurrence of the first event; 
 generate, based on combining the plurality of event data fields and the timing information, a second event of a discoverable event stream corresponding to the pre-processed event stream; and 
 generate, within a time series of the discoverable event stream, based on the timing information and an event field of the plurality of event data fields, a time series entry corresponding to the second event. 
   
     
     
         18 . The apparatus of  claim 17 , wherein to determine timing information, the processor is further configured to:
 determine the timing information based on a plurality of events received over the pre-processed event stream.   
     
     
         19 . The apparatus of  claim 17 , wherein to determine timing information, the processor is further configured to:
 identify a timestamp generated by a client application within the first event;   determine a correction value based upon a local system clock and the timestamp; and   determine the timing information based upon the correction value.   
     
     
         20 . The apparatus of  claim 17 , wherein to determine timing information, the processor is further configured to:
 identify a highest timestamp amongst a plurality of events of the pre-processed event stream;   determine a correction value based upon a local system clock and highest timestamp; and   determine the timing information based upon the correction value.   
     
     
         21 . The apparatus of  claim 20 , wherein to determine the correction value, the processor is further configured to subtract the highest timestamp from the local system clock to determine the correction value. 
     
     
         22 . The apparatus of  claim 19 , wherein to generate the second event of the discoverable event stream corresponding to the pre-processed event stream, the processor is further configured to:
 generate one or more common data fields based at least in part on the plurality of event data fields, the one or more common data fields shared across a plurality of discoverable event streams having different data sources; and   generate, based on combining the plurality of event data fields and the one or more common data fields, the second event of the discoverable event stream corresponding to the pre-processed event stream.   
     
     
         23 . The apparatus of  claim 22 , wherein the one or more common data fields include an event identifier identifying an event, a name of the event, a first timestamp indicating the time of occurrence of the event, a source identifier identifying a source of the event, a second timestamp indicating an arrival time of the event data corresponding to the event at an ingestion component, and/or a tenant identifier identifying a tenant of a multi-tenant system associated with the event. 
     
     
         24 . A non-transitory computer-readable device having instructions thereon that, when executed by at least one computing device, causes the at least one computing device to perform operations comprising:
 receiving a first event of a pre-processed event stream, the first event including a plurality of event data fields representing an occurrence of the first event;   determining timing information based on the plurality of event data fields representing the occurrence of the first event;   generating, based on combining the plurality of event data fields and the timing information, a second event of a discoverable event stream corresponding to the pre-processed event stream; and   generating, within a time series of the discoverable event stream, based on the timing information and an event field of the plurality of event data fields, a time series entry corresponding to the second event.   
     
     
         25 . The non-transitory computer-readable device of  claim 24 , wherein determining the timing information comprises:
 determining the timing information based on a plurality of events received over the pre-processed event stream.   
     
     
         26 . The non-transitory computer-readable device of  claim 24 , wherein determining the timing information comprises:
 identifying a timestamp generated by a client application within the first event;   determining a correction value based upon a local system clock and the timestamp; and   determining the timing information based upon the correction value.   
     
     
         27 . The non-transitory computer-readable device of  claim 24 , wherein determining the timing information comprises:
 identifying a highest timestamp amongst a plurality of events of the pre-processed event stream;   determining a correction value based upon a local system clock and highest timestamp; and   determining the timing information based upon the correction value.   
     
     
         28 . The non-transitory computer-readable device of  claim 24 , wherein generating the second event of the discoverable event stream corresponding to the pre-processed event stream, comprises:
 generating one or more common data fields based at least in part on the plurality of event data fields, the one or more common data fields shared across a plurality of discoverable event streams having different data sources; and   generating, based on combining the plurality of event data fields and the one or more common data fields, the second event of the discoverable event stream corresponding to the pre-processed event stream.   
     
     
         29 . The non-transitory computer-readable device of  claim 28 , wherein the one or more common data fields include an event identifier identifying an event, a name of the event, a first timestamp indicating the time of occurrence of the event, a source identifier identifying a source of the event, a second timestamp indicating an arrival time of the event of the event data corresponding to the event at an ingestion component, and/or a tenant identifier identifying a tenant of a multi-tenant system associated with the event.