Continuous feedback-controlled deployment of message transforms in a distributed messaging system

Abstract

The present invention solves the disadvantages of the prior art and provides a distributed messaging system supporting stateful subscriptions. A stateful publish-subscribe system extends the functionality of the content-based approach to include more general state-valued expressions. A plurality of broker machines is provided to deliver messages sent by publishing clients toward subscribing clients based upon the contents of the messages and stateful transformations requested by the subscribing clients. Subscription specifications are analyzed by a compiler and converted into a collection of message transforms and views. A deployment service component deploys tasks to optimize system performance. A monitoring services component detects a possible need to reconfigure. A performance optimization service component computes new assignment of transforms. A continuous deployment service implements a redeployment protocol that installs changes to transform placement while the existing publish-subscribe system continues to operate.

1 . A method for moving a message transform from a source broker to a target broker in a distributed messaging system, wherein the distributed messaging system includes a network of broker machines, wherein a plurality of message transforms are deployed in the network of broker machines, and wherein the plurality of message transforms take as input one or more sequences of messages and produce as output one or more sequences of messages, the method comprising: 
 starting on the target broker a copy of a given message transform from the source broker;    duplicating all inputs into the given message transform on the source broker to additionally go to the copy of the given message transform on the target broker; and    duplicating all outputs of the given message transform on the source broker to also go from the copy of the given message transform on the target broker to appropriate children nodes within the distributed messaging system.    
   
   
       2 . The method of  claim 1 , further comprising: 
 copying a state of the given message transform from the source broker to the copy of the given message transform on the target broker;    
   
   
       3 . The method of  claim 1 , further comprising: 
 terminating the given message transform on the source broker.    
   
   
       4 . The method of  claim 3 , wherein terminating the given transform on the source broker is performed responsive to the copy of the given message transform passing a self-diagnostic.  
   
   
       5 . The method of  claim 1 , further comprising: 
 receiving a plurality of input messages associated with input streams;    applying the copy of the given transform at the target broker to the plurality of input messages to form an output message, wherein the output message depends on at least two input messages from the plurality of input messages; and    delivering the output message in an output stream.    
   
   
       6 . A method for continuous deployment of message transforms in a distributed messaging system, wherein the distributed messaging system includes a network of broker machines, the method comprising: 
 deploying a plurality of message transforms in the network of broker machines, wherein the plurality of message transforms take as input one or more sequences of messages and produce as output one or more sequences of messages;    receiving performance information from the network of broker machines; and    redeploying at least a subset of the plurality of message transforms in the network of broker machines based on the performance information.    
   
   
       7 . The method of  claim 6 , wherein redeploying at least a subset of the plurality of message transforms includes sending one or more broker instructions to at least one broker machine in the network of broker machines.  
   
   
       8 . The method of  claim 6 , wherein redeploying at least a subset of the plurality of message transforms includes moving a given message transform from a source broker to a target broker in the distributed messaging system.  
   
   
       9 . The method of  claim 8 , wherein moving a given message transform includes: 
 starting on the target broker a copy of the given message transform from the source broker;    duplicating all inputs into the given message transform on the source broker to additionally go to the copy of the given message transform on the target broker; and    duplicating all outputs of the given message transform on the source broker to also go from the copy of the given message transform on the target broker to appropriate children nodes within the distributed messaging system.    
   
   
       10 . An apparatus for implementing a stateful subscription service, the apparatus comprising: 
 an overlay network including a plurality of broker machines;    a plurality of publishing clients that publish messages to published message streams;    a performance optimization service that determines an assignment of message transforms to broker machines in the overlay network; and    a continuous deployment service that deploys the assignment of message transforms to the plurality of broker machines.    
   
   
       11 . The apparatus of  claim 10 , further comprising: 
 a plurality of execution engines running on the plurality of broker machines, wherein each execution engine within the plurality of execution engine receives input messages, processes the input messages using the transform objects, and routes output messages toward subscribers.    
   
   
       12 . The apparatus of  claim 11 , wherein at least one message transform in the assignment of message transforms has at least one message delivered to its output stream that depends on more than one input message from an input stream of the message transform.  
   
   
       13 . An apparatus for moving a message transform from a source broker to a target broker in a distributed messaging system, wherein the distributed messaging system includes a network of broker machines, wherein a plurality of message transforms are deployed in the network of broker machines, and wherein the plurality of message transforms take as input one or more sequences of messages and produce as output one or more sequences of messages, the apparatus comprising: 
 means for starting on the target broker a copy of a given message transform from the source broker;    means for duplicating all inputs into the given message transform on the source broker to additionally go to the copy of the given message transform on the target broker; and    means for duplicating all outputs of the given message transform on the source broker to also go from the copy of the given message transform on the target broker to appropriate children nodes within the distributed messaging system.    
   
   
       14 . The apparatus of  claim 13 , further comprising: 
 means for copying a state of the given message transform from the source broker to the copy of the given message transform on the target broker;    
   
   
       15 . The apparatus of  claim 13 , further comprising: 
 means for terminating the given message transform on the source broker.    
   
   
       16 . The apparatus of  claim 15 , wherein the means for terminating the given transform on the source broker terminates the given transform responsive to the copy of the given message transform passing a self-diagnostic.  
   
   
       17 . The apparatus of  claim 13 , further comprising: 
 means for receiving a plurality of input messages associated with input streams;    means for applying the copy of the given transform at the target broker to the plurality of input messages to form an output message, wherein the output message depends on at least two input messages from the plurality of input messages; and    means for delivering the output message in an output stream.    
   
   
       18 . An apparatus for continuous deployment of message transforms in a distributed messaging system, wherein the distributed messaging system includes a network of broker machines, the apparatus comprising: 
 means for deploying a plurality of message transforms in the network of broker machines, wherein the plurality of message transforms take as input one or more sequences of messages and produce as output one or more sequences of messages;    means for receiving performance information from the network of broker machines; and    means for redeploying at least a subset of the plurality of message transforms in the network of broker machines based on the performance information.    
   
   
       19 . The apparatus of  claim 18 , wherein the means for redeploying at least a subset of the plurality of message transforms includes means for sending one or more broker instructions to at least one broker machine in the network of broker machines.  
   
   
       20 . The apparatus of  claim 18 , wherein the means for redeploying at least a subset of the plurality of message transforms includes means for moving a given message transform from a source broker to a target broker in the distributed messaging system.  
   
   
       21 . The apparatus of  claim 20 , wherein the means for moving a given message transform includes: 
 means for starting on the target broker a copy of the given message transform from the source broker;    means for duplicating all inputs into the given message transform on the source broker to additionally go to the copy of the given message transform on the target broker; and    means for duplicating all outputs of the given message transform on the source broker to also go from the copy of the given message transform on the target broker to appropriate children nodes within the distributed messaging system.    
   
   
       22 . A computer program product, in a computer readable medium, for moving a message transform from a source broker to a target broker in a distributed messaging system, wherein the distributed messaging system includes a network of broker machines, wherein a plurality of message transforms are deployed in the network of broker machines, and wherein the plurality of message transforms take as input one or more sequences of messages and produce as output one or more sequences of messages, the computer program product comprising: 
 instructions for starting on the target broker a copy of a given message transform from the source broker;    instructions for duplicating all inputs into the given message transform on the source broker to additionally go to the copy of the given message transform on the target broker; and    instructions for duplicating all outputs of the given message transform on the source broker to also go from the copy of the given message transform on the target broker to appropriate children nodes within the distributed messaging system.    
   
   
       23 . A computer program product, in a computer readable medium, for continuous deployment of message transforms in a distributed messaging system, wherein the distributed messaging system includes a network of broker machines, the computer program product comprising: 
 instructions for deploying a plurality of message transforms in the network of broker machines, wherein the plurality of message transforms take as input one or more sequences of messages and produce as output one or more sequences of messages;    instructions for receiving performance information from the network of broker machines; and    instructions for redeploying at least a subset of the plurality of message transforms in the network of broker machines based on the performance information.