Architecture for large payload handling in event pipeline

Abstract

Systems and methods are provided for automatically orchestrating the handling of events through a processing pipeline without limitation (or without a substantial limitation) as to the size of the event payload associated with the event. The event pipeline system stores event payloads in data stores and generates notifications regarding the events. The notifications may be placed into event streams for handling by various processing components of the event pipeline system. The processing components may receive notifications or events that they are to process, and may separately access event payloads from the data stores. The processing components may generate and save processed event payloads to the data stores in a streaming fashion such that the computing resources of the processing components do not limit (or substantially limit) the size of the event payloads that the processing components may handle.

1 .- 20 . (canceled) 
     
     
         21 . A computer-implemented method comprising:
 under control of a computing system comprising one or more computing devices configured to execute specific instructions,
 obtaining a source event notification from a notification data store of a source event stream; 
 determining to generate a transformed event from a source event associated with the source event notification; 
 extracting, from the source event notification, a source event payload storage location of an event payload data store associated with the source event stream; 
 obtaining a source event payload from the source event payload storage location; 
 generating a transformed event payload from the source event payload; 
 storing the transformed event payload to a transformed event payload storage location of an event payload data store of a target event stream; and 
 sending a transformed event notification to an event notification data store associated with the target event stream, wherein the transformed event notification references the transformed event payload storage location. 
   
     
     
         22 . The computer-implemented method of  claim 21 , wherein sending the transformed event notification to the event notification data store comprises sending the transformed event notification to a first computing device that is different from a second computing device at which the transformed event payload storage location is located. 
     
     
         23 . The computer-implemented method of  claim 21 , wherein sending the transformed event notification to the event notification data store comprises sending the transformed event notification to an event notification queue. 
     
     
         24 . The computer-implemented method of  claim 21 , further comprising determining the transformed event payload storage location, wherein the transformed event payload storage location comprises a path of a data object in an object data store. 
     
     
         25 . The computer-implemented method of  claim 21 , wherein generating the transformed event payload from the source event payload comprises applying, to the source event payload, at least one of a formatting transformation or a content transformation. 
     
     
         26 . The computer-implemented method of  claim 21 , wherein generating the transformed event payload from the source event payload comprises applying, to the source event payload, at least one of a filter or an augmentation. 
     
     
         27 . The computer-implemented method of  claim 21 , further comprising determining the transformed event payload storage location, wherein the transformed event payload storage location comprises a path of a data object in an object data store. 
     
     
         28 . The computer-implemented method of  claim 21 , further comprising:
 obtaining a subscribed event notification from a notification data store of a subscribed event stream;   extracting a subscribed event payload storage location from the subscribed event notification;   obtaining a subscribed event payload from the subscribed event payload storage location of an event payload data store of the subscribed event stream; and   providing the subscribed event payload to an event subscriber device.   
     
     
         29 . The computer-implemented method of  claim 21 , further comprising:
 determining to generate one of a batched event or a joined event from a plurality of events satisfying a selection criterion; and   for each event of the plurality of events satisfying a selection criterion:
 obtaining a corresponding event notification from a corresponding notification data store; 
 extracting, from the corresponding event notification, location data for a corresponding event payload in a corresponding event payload data store; 
 obtaining the corresponding event payload from the corresponding event payload data store; and 
 storing, at a storage location in a second event payload data store associated with a second event stream, at least a portion of the corresponding event payload. 
   
     
     
         30 . The computer-implemented method of  claim 29 , further comprising applying a transformation to the corresponding event payload prior to storing the portion of the corresponding event payload. 
     
     
         31 . A system comprising:
 computer-readable memory storing executable instructions; and   one or more processors in communication with the computer-readable memory and programmed by the executable instructions to at least:
 obtain a source event notification from a notification data store of a source event stream; 
 determine to generate a transformed event from a source event associated with the source event notification; 
 extract, from the source event notification, a source event payload storage location of an event payload data store associated with the source event stream; 
 obtain a source event payload from the source event payload storage location; 
 generate a transformed event payload from the source event payload; 
 store the transformed event payload to a transformed event payload storage location of an event payload data store of a target event stream; and 
 send a transformed event notification to an event notification data store associated with the target event stream, wherein the transformed event notification references the transformed event payload storage location. 
   
     
     
         32 . The system of  claim 31 , wherein to send the transformed event notification to the event notification data store, the one or more processors are further programmed by the executable instructions to send the transformed event notification to a first computing device that is different from a second computing device at which the transformed event payload storage location is located. 
     
     
         33 . The system of  claim 31 , wherein to send the transformed event notification to the event notification data store, the one or more processors are further programmed by the executable instructions to send the transformed event notification to an event notification queue. 
     
     
         34 . The system of  claim 31 , wherein the one or more processors are further programmed by the executable instructions to determine the transformed event payload storage location, wherein the transformed event payload storage location comprises a path of a data object in an object data store. 
     
     
         35 . The system of  claim 31 , wherein to generate the transformed event payload from the source event payload, the one or more processors are further programmed by the executable instructions to apply, to the source event payload, at least one of a formatting transformation or a content transformation. 
     
     
         36 . The system of  claim 31 , wherein to generate the transformed event payload from the source event payload, the one or more processors are further programmed by the executable instructions to apply, to the source event payload, at least one of a filter or an augmentation. 
     
     
         37 . The system of  claim 31 , wherein the one or more processors are further programmed by the executable instructions to determine the transformed event payload storage location, wherein the transformed event payload storage location comprises a path of a data object in an object data store. 
     
     
         38 . The system of  claim 31 , wherein the one or more processors are further programmed by the executable instructions to at least:
 obtain a subscribed event notification from a notification data store of a subscribed event stream;   extract a subscribed event payload storage location from the subscribed event notification;   obtain a subscribed event payload from the subscribed event payload storage location of an event payload data store of the subscribed event stream; and   provide the subscribed event payload to an event subscriber device.   
     
     
         39 . The system of  claim 31 , wherein the one or more processors are further programmed by the executable instructions to:
 determine to generate one of a batched event or a joined event from a plurality of events satisfying a selection criterion; and   for each event of the plurality of events satisfying a selection criterion:
 obtain a corresponding event notification from a corresponding notification data store; 
 extract, from the corresponding event notification, location data for a corresponding event payload in a corresponding event payload data store; 
 obtain the corresponding event payload from the corresponding event payload data store; and 
 store, at a storage location in a second event payload data store associated with a second event stream, at least a portion of the corresponding event payload. 
   
     
     
         40 . The system of  claim 39 , wherein the one or more processors are further programmed by the executable instructions to apply a transformation to the corresponding event payload prior to storing the portion of the corresponding event payload.