Data replication in scalable messaging system

Abstract

A plurality of channels are maintained, wherein each channel comprises a plurality of messages stored in one or more buffers, wherein a first node stores a first copy of a first buffer for a first channel of the plurality of channels, wherein each of a plurality of secondary nodes stores a respective second copy of the first buffer, and wherein at least one subscriber is subscribed to messages of the first channel. A failure of the first node is detected. One of the plurality of secondary nodes are identified as a second node to access a second copy of the first buffer. The at least one subscriber is redirected from the first node to the second node and messages stored in the second copy on the second node are retrieved for the at least one subscriber.

What is claimed is: 
     
         1 . A method, comprising:
 maintaining a plurality of channels,
 wherein each channel comprises a plurality of messages stored in one or more buffers, 
 wherein a first node stores a first copy of a first buffer for a first channel of the plurality of channels, 
 wherein each of a plurality of secondary nodes stores a respective second copy of the first buffer, and 
 wherein at least one subscriber is subscribed to messages of the first channel; 
   detecting, by one or more computer processors, a failure of the first node;   identifying, by the one or more computer processors, one of the plurality of secondary nodes as a second node to access a second copy of the first buffer;   redirecting, by the one or more computer processors, the at least one subscriber from the first node to the second node; and   retrieving messages stored in the second copy on the second node for the at least one subscriber.   
     
     
         2 . The method of  claim 1 , wherein the messages for each channel are stored in the one or more buffers according to an order. 
     
     
         3 . The method of  claim 1 , wherein each buffer comprises a respective time-to-live. 
     
     
         4 . The method of  claim 1 , wherein identifying one of the plurality of secondary nodes as the second node to access the second copy of the first buffer comprises:
 selecting the second node based on a respective workload for each node of the plurality of secondary nodes.   
     
     
         5 . The method of  claim 1 , comprising:
 redirecting at least one publisher from the first node to the second node; and   sending messages from the at least one publisher to the second node for storage in the second copy.   
     
     
         6 . The method of  claim 1 , wherein the respective second copies of the first buffer comprise a chain of copies including at least a head copy and a tail copy. 
     
     
         7 . The method of  claim 6 , comprising:
 forwarding messages stored in a particular copy of the chain of copies to a next adjacent copy of the chain of copies for storage, until messages in the head copy of the chain of copies are stored in the tail copy of the chain of copies.   
     
     
         8 . The method of  claim 1 , wherein detecting the failure of the first node comprises:
 detecting failure of the at least one subscriber obtaining messages from the first copy.   
     
     
         9 . The method of  claim 1 , wherein at least one publisher sends messages of the first buffer to the first node for storage in the first copy before being redirected to the second node. 
     
     
         10 . The method of  claim 1 , wherein detecting the failure of the first node comprises:
 detecting failure of at least one publisher sending messages to the first node for storage in the first copy.   
     
     
         11 . A system, comprising:
 one or more computer processors programmed to perform operations to:
 maintain a plurality of channels,
 wherein each channel comprises a plurality of messages stored in one or more buffers, 
 wherein a first node stores a first copy of a first buffer for a first channel of the plurality of channels, 
 wherein each of a plurality of secondary nodes stores a respective second copy of the first buffer, and 
 wherein at least one subscriber is subscribed to messages of the first channel; 
 
 detect a failure of the first node; 
 identify one of the plurality of secondary nodes as a second node to access a second copy of the first buffer; 
 redirect the at least one subscriber from the first node to the second node; and 
 retrieve messages stored in the second copy on the second node for the at least one subscriber. 
   
     
     
         12 . The system of  claim 11 , wherein the messages for each channel are stored in the one or more buffers according to an order. 
     
     
         13 . The system of  claim 11 , wherein to identify one of the plurality of secondary nodes as the second node to access the second copy of the first buffer the one or more computer processors are further to:
 select the second node based on a respective workload for each node of the plurality of secondary nodes.   
     
     
         14 . The system of  claim 11 , wherein the operations are further to:
 redirect at least one publisher from the first node to the second node; and   send messages from the at least one publisher to the second node for storage in the second copy.   
     
     
         15 . The system of  claim 11 , wherein the respective second copies of the first buffer comprise a chain of copies including at least a head copy and a tail copy. 
     
     
         16 . The system of  claim 15 , wherein the operations are further to:
 forward messages stored in a particular copy of the chain of copies to a next adjacent copy of the chain of copies for storage, until messages in the head copy of the chain of copies are stored in the tail copy of the chain of copies.   
     
     
         17 . The system of  claim 11 , wherein to detect the failure of the first node the one or more computer processors are further to:
 detect failure of the at least one subscriber obtaining messages from the first copy.   
     
     
         18 . The system of  claim 11 , wherein at least one publisher sends messages of the first buffer to the first node for storage in the first copy before being redirected to the second node. 
     
     
         19 . The system of  claim 11 , wherein to detect the failure of the first node the one or more computer processors are further to:
 detect failure of at least one publisher sending messages to the first node for storage in the first copy.   
     
     
         20 . A non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more computer processors, cause the one or more computer processors to:
 maintain a plurality of channels,
 wherein each channel comprises a plurality of messages stored in one or more buffers, 
 wherein a first node stores a first copy of a first buffer for a first channel of the plurality of channels, 
 wherein each of a plurality of secondary nodes stores a respective second copy of the first buffer, and 
 wherein at least one subscriber is subscribed to messages of the first channel; 
   detect a failure of the first node;   identify one of the plurality of secondary nodes as a second node to access a second copy of the first buffer;   redirect the at least one subscriber from the first node to the second node; and   
       retrieve messages stored in the second copy on the second node for the at least one subscriber.