In-flight transformations over a data stream

Abstract

Disclosed herein are system, method, and computer program product embodiments for performing in-flight transformations of data streams. Complex event processing (CEP) operating on event records from various data streams can operate by storing the streamed event record data in a database, and then querying the data to perform a data transformation. However, in order to improve the performance of the data transformations and streamline CEP, a serverless architecture is introduced that can perform data transformations directly on streams, using attributes of the streamed record data defined in a schema. The resulting transformed data can then be provided by the serverless architecture to the CEP for direct access of the data most needed by the CEP.

What is claimed is: 
     
       1. A computer implemented method, comprising:
 connecting, by one or more computing devices, a plurality of nodes within a cluster to a data stream conforming to a schema, each of the plurality of nodes is configured to receive each of a plurality of data stream segments segmented based on a number of an event record from the data stream; provisioning, by the one or more computing devices, the plurality of nodes to perform a data transformation on elements of the plurality of data stream segments as defined by the schema, the data transformation produces transformed data by performing a query defined by the schema of the cluster where the plurality of nodes exists, the query is performed directly through a cloud service before the data stream is directly received by a data sink, and the schema defines a data organization of the plurality of data stream segments to include client ID, date, merchant, and amount corresponding to the event record; and receiving, by the one or more computing devices, the transformed data at the data sink connected to the plurality of nodes. 
 
     
     
       2. The computer implemented method of  claim 1 , wherein a complex events processing system executing on the one or more computing devices is configured to act as the data sink and to perform an action based on the transformed data. 
     
     
       3. The computer implemented method of  claim 1 , wherein the query comprises an operation performed on the elements of a data stream segment and on additional elements of an additional data stream segment. 
     
     
       4. The computer implemented method of  claim 1 , further comprising:
 obtaining, by the one or more computing devices, load requirement information of the data stream; and 
 notifying, by the one or more computing devices, the cluster to adjust node availability based on load requirements of the data stream. 
 
     
     
       5. The computer implemented method of  claim 1 , further comprising:
 creating, by the one or more computing devices, a configuration file specifying configuration information for the plurality of nodes and one or more additional nodes within the cluster. 
 
     
     
       6. The computer implemented method of  claim 1 , wherein the plurality of nodes is configured as a function as a service (FaaS) consumer. 
     
     
       7. The computer implemented method of  claim 1 , wherein the event record is a record of a purchase event made by a customer using a credit card. 
     
     
       8. A system, comprising:
 a memory configured to store operations; and 
 one or more processors configured to perform the operations, the operations comprising:
 connecting a plurality of nodes within a cluster to a data stream conforming to a schema, wherein each of the plurality of nodes is configured to receive each of a plurality of data stream segments segmented based on a number of an event record from the data stream, 
 provisioning the plurality of nodes to perform a data transformation on elements of the plurality of data stream segments as defined by the schema, wherein the data transformation produces transformed data by performing a query defined by the schema of the cluster where the plurality of nodes exists, the query is performed directly through a cloud service before the data stream is directly received by a data sink, and the schema defines a data organization of the plurality of data stream segments to include client ID, date, merchant, and amount corresponding to the event record; and 
 receiving the transformed data at the data sink connected to the plurality of nodes. 
 
 
     
     
       9. The system of  claim 8 , wherein a complex events processing system executing on the one or more processors is configured to act as the data sink and to perform an action based on the transformed data. 
     
     
       10. The system of  claim 8 , wherein the query comprises an operation performed on the elements of a data stream segment and on additional elements of an additional data stream segment. 
     
     
       11. The system of  claim 8 , the operations further comprising:
 obtaining load requirement information of the data stream; and 
 notifying the cluster to adjust node availability based on load requirements of the data stream. 
 
     
     
       12. The system of  claim 8 , the operations further comprising:
 creating a configuration file specifying configuration information for the plurality of nodes and one or more additional nodes within the cluster. 
 
     
     
       13. The system of  claim 8 , wherein the plurality of nodes is configured as a function as a service (FaaS) consumer. 
     
     
       14. The system of  claim 8 , wherein the event record is a record of a purchase event made by a customer using a credit card. 
     
     
       15. A computer readable storage device having instructions stored thereon, execution of which, by one or more processing devices, causes the one or more processing devices to perform operations comprising:
 connecting a plurality of nodes within a cluster to a data stream conforming to a schema, wherein each of the plurality of nodes is configured to receive each of a plurality of data stream segments segmented based on a number of an event record from the data stream; 
 provisioning the plurality of nodes to perform a data transformation on elements of the plurality of data stream segments as defined by the schema, wherein the data transformation produces transformed data by performing a query defined by the schema of the cluster where the plurality of nodes exists, the query is performed directly through a cloud service before the data stream is directly received by a data sink, and the schema defines a data organization of the plurality of data stream segments to include client ID, date, merchant, and amount corresponding to the event record; and 
 receiving the transformed data at the data sink connected to the plurality of nodes. 
 
     
     
       16. The computer readable storage device of  claim 15 , wherein a complex events processing system executing on the one or more processing devices is configured to act as the data sink and to perform an action based on the transformed data. 
     
     
       17. The computer readable storage device of  claim 16 , wherein the query comprises an operation performed on the elements of a data stream segment and on additional elements of an additional data stream segment. 
     
     
       18. The computer readable storage device of  claim 15 , the operations further comprising:
 obtaining load requirement information of the data stream; and 
 notifying the cluster to adjust node availability based on load requirements of the data stream. 
 
     
     
       19. The computer readable storage device of  claim 15 , the operations further comprising:
 creating a configuration file specifying configuration information for the plurality of nodes and one or more additional nodes within the cluster. 
 
     
     
       20. The computer readable storage device of  claim 15 , wherein the event record a record of a purchase event made by a customer using a credit card.