System and method for minimizing latency in data consumption system

Abstract

A campaign server is disclosed which includes a multi-level queue architecture that isolates campaign processing from the delays and inconsistencies of target population information retrieval and a data streaming service system with latency and error remediation measures are disclosed, where the campaign server and the data consumption system can be part of a consolidated system. The data consumption system may include various components to process one or more offers to one or more customers and component configured to detect an error associated the preparation of the errors and/or ii) initiate a retry attempt to process the one or more offers, and and/or storing an instruction in a storage component of the streaming data system to permanently prohibit an attempt to generate the one or more offers at a subsequent time if the retry attempt expires without processing the one or more offers.

1 .- 20 . (canceled) 
     
     
         21 . A computer-implemented method, comprising:
 generating, by a processor, a plurality of offers for one or more customers based on customer information received from a plurality of partitions associated with a topic of a streaming component of a streaming data system;   storing, by the processor, a first offer of the plurality of offers in a first queue for retrieval by a second queue associated with the one or more customers;   processing, by a messaging agent executing on the processor, the first offer stored in the second queue;   selecting, by the processor, a threshold number of records based on one or more delays in the streaming data system, an available network bandwidth of the streaming data system, or a size of records stored by the first and second queues;   determining, by the processor, the first queue stores a number of records equaling the threshold number of records; and   transmitting, by the processor, an instruction to the streaming component specifying to stop transmitting additional customer information to be stored in the first queue.   
     
     
         22 . The computer-implemented method of  claim 21 , wherein the plurality of offers are generated based on a plurality of operations of a process flow, the method further comprising:
 detecting, by the processor at a first operation of the plurality of operations of the process flow, an error associated with generating a second offer of the plurality of offers; and   initiating, by the processor, a retry attempt to generate the second offer, wherein the retry attempt begins at the first operation of the process flow.   
     
     
         23 . The computer-implemented method of  claim 22 , further comprising:
 determining, by the processor, that the retry attempt expires without processing the second offer; and   transmitting, by the processor, an instruction to a plurality of processing components of the streaming data system to permanently prohibit a subsequent attempt to generate the second offer and store the second offer in the first or second queues at a subsequent time.   
     
     
         24 . The computer-implemented method of  claim 21 , further comprising:
 retrieving, by the processor, one or more rules that govern creation of the plurality of offers;   retrieving, by the processor, one or more templates for the plurality of offers based on receipt of a rules instruction; and   transmitting, by the processor, the templates to the one or more customers using one or more of a plurality of channels.   
     
     
         25 . The computer-implemented method of  claim 21 , wherein the instruction is transmitted to a plurality of processing components of the streaming component, wherein the plurality of processing components are configured to perform a plurality of processing operations of a process flow to generate the plurality of offers. 
     
     
         26 . The computer-implemented method of  claim 25 , wherein the customer information is provided to the plurality of processing components to perform the plurality of processing operations of the process flow in parallel such that an error associated with preparation of a second offer of the plurality of offers based on a first partition of the plurality of partitions does not impact processing of a third offer of the plurality of offers based on a second partition of the plurality of partitions. 
     
     
         27 . The computer-implemented method of  claim 26 , further comprising:
 transmitting another instruction to the plurality of processing components to cause the plurality of processing components to refrain from subsequently attempting to generate the second offer.   
     
     
         28 . A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a processor, cause the processor to:
 generate a plurality of offers for one or more customers based on customer information received from a plurality of partitions associated with a topic of a streaming component of a streaming data system;   store a first offer of the plurality of offers in a first queue for retrieval by a second queue associated with the one or more customers;   process, by a messaging agent, the first offer stored in the second queue;   select a threshold number of records based on one or more delays in the streaming data system, an available network bandwidth of the streaming data system, or a size of records stored by the first and second queues;   determine the first queue stores a number of records equaling the threshold number of records; and   transmit an instruction to the streaming component specifying to stop transmitting additional customer information to be stored in the first queue.   
     
     
         29 . The computer-readable storage medium of  claim 28 , wherein the plurality of offers are generated based on a plurality of operations of a process flow, wherein the instructions further cause the processor to:
 detect, at a first operation of the plurality of operations of the process flow, an error associated with generating a second offer of the plurality of offers; and   initiate a retry attempt to generate the second offer, wherein the retry attempt begins at the first operation of the process flow.   
     
     
         30 . The computer-readable storage medium of  claim 29 , wherein the instructions further cause the processor to:
 determine that the retry attempt expires without processing the second offer; and   transmit an instruction to a plurality of processing components of the streaming data system to permanently prohibit a subsequent attempt to generate the second offer and store the second offer in the first or second queues at a subsequent time.   
     
     
         31 . The computer-readable storage medium of  claim 28 , wherein the instructions further cause the processor to:
 retrieve one or more rules that govern creation of the plurality of offers;   retrieve one or more templates for the plurality of offers based on receipt of a rules instruction; and   transmit the templates to the one or more customers using one or more of a plurality of channels.   
     
     
         32 . The computer-readable storage medium of  claim 28 , wherein the instruction is transmitted to a plurality of processing components of the streaming component, wherein the plurality of processing components are configured to perform a plurality of processing operations of a process flow to generate the plurality of offers. 
     
     
         33 . The computer-readable storage medium of  claim 32 , wherein the customer information is provided to the plurality of processing components to perform the plurality of processing operations of the process flow in parallel such that an error associated with preparation of a second offer of the plurality of offers based on a first partition of the plurality of partitions does not impact processing of a third offer of the plurality of offers based on a second partition of the plurality of partitions. 
     
     
         34 . The computer-readable storage medium of  claim 33 , wherein the instructions further cause the processor to:
 transmit another instruction to the plurality of processing components to cause the plurality of processing components to refrain from subsequently attempting to generate the second offer.   
     
     
         35 . A computing apparatus comprising:
 a processor; and   a memory storing instructions that, when executed by the processor, cause the processor to:   generate a plurality of offers for one or more customers based on customer information received from a plurality of partitions associated with a topic of a streaming component of a streaming data system;   store a first offer of the plurality of offers in a first queue for retrieval by a second queue associated with the one or more customers;   process, by a messaging agent, the first offer stored in the second queue;   select a threshold number of records based on one or more delays in the streaming data system, an available network bandwidth of the streaming data system, or a size of records stored by the first and second queues;   determine the first queue stores a number of records equaling the threshold number of records; and   transmit an instruction to the streaming component specifying to stop transmitting additional customer information to be stored in the first queue.   
     
     
         36 . The computing apparatus of  claim 35 , wherein the plurality of offers are generated based on a plurality of operations of a process flow, wherein the instructions further cause the processor to:
 detect, at a first operation of the plurality of operations of the process flow, an error associated with generating a second offer of the plurality of offers; and   initiate a retry attempt to generate the second offer, wherein the retry attempt begins at the first operation of the process flow.   
     
     
         37 . The computing apparatus of  claim 36 , wherein the instructions further cause the processor to:
 determine that the retry attempt expires without processing the second offer; and   transmit an instruction to a plurality of processing components of the streaming data system to permanently prohibit a subsequent attempt to generate the second offer and store the second offer in the first or second queues at a subsequent time.   
     
     
         38 . The computing apparatus of  claim 35 , wherein the instructions further cause the processor to:
 retrieve, by the second queue, the first offer from the first queue;   pull, by the messaging agent, the first offer from the second queue;   determine the first offer is successfully provided to the one or more customers by the messaging agent via one or more of a plurality of channels; and   update the topic of the streaming component with information associated with the successful provision of the first offer.   
     
     
         39 . The computing apparatus of  claim 35 , wherein the instruction is transmitted to a plurality of processing components of the streaming component, wherein the plurality of processing components are configured to perform a plurality of processing operations of a process flow to generate the plurality of offers. 
     
     
         40 . The computing apparatus of  claim 39 , wherein the customer information is provided to the plurality of processing components to perform the plurality of processing operations of the process flow in parallel such that an error associated with preparation of a second offer of the plurality of offers based on a first partition of the plurality of partitions does not impact processing of a third offer of the plurality of offers based on a second partition of the plurality of partitions.