Failure recovery in a fault tolerant architecture

Abstract

A system/process provides fault tolerance to an integrated component system which integrates core components of a transaction processing system into a single processing platform, i.e., a single server, enabling elimination of the network interconnects and associated latencies introduced thereby in favor of much faster interconnects, such as inter-process communication and shared memory communication messaging, where a failure of any one component necessitates failing over the entire system to a backup thereof.

We claim: 
     
       1. A system comprising:
 a plurality of transaction processing servers coupled with each other via an electronic communications network, each of the plurality of transaction processing servers comprising a processor which implements a plurality of components coupled with each other, each of the plurality of components characterized by a current state and configured to perform one or more of receiving an electronic message, processing the received electronic message and generating an electronic result message indicative of a result of an operation which may alter the current state thereof, wherein one of the plurality of transaction processing servers is designated as a primary instance and each other of the plurality of transaction processing servers is designated as a backup instance, wherein each of the plurality of transaction processing servers is operative to receive an incoming request for a transaction, process the received incoming request to identify a previously received request for a transaction counter thereto in an attempt to satisfy one or both of the incoming request and the previously received request and generate a result indicative thereof, the transaction processing server designated as the primary instance at least commencing processing of the incoming request prior to each of the other of the plurality of transaction processing servers designated as the backup instance, wherein one or more of the plurality of components of the transaction processing server designated as the primary instance are each configured to communicate the electronic result message indicative of the result of the operation thereof to a corresponding component of each of the other of the plurality of transaction processing servers designated as the backup instance; and 
 a plurality of fault tolerance processors, each of the plurality of transaction processing servers coupled with one of the plurality of fault tolerance processors, each of the plurality of fault tolerance processors configured to determine that a failure has occurred in the transaction processing server designated as the primary instance and, subsequent thereto, cause one of the other of the plurality of transaction processing servers to be designated as a new primary instance, wherein the one of other of the plurality of transaction processing servers which is designated as the new primary instance is configured to synchronize the current state of each of the plurality of components thereof based on received electronic messages indicative of the result of the operation of the corresponding component of the primary instance, and process any incoming requests not processed by the primary instance before commencing processing of new incoming requests. 
 
     
     
       2. The system of  claim 1 , wherein upon receipt of the electronic result message indicative of the result of the operation of the one or more of the plurality of components of the primary instance, the corresponding component of each of the other of the plurality of transaction processing servers designated as the backup instance compares a result of the operation thereof with the result indicated by the received electronic result message to determine whether one of the plurality of components of the one of the plurality of transaction processing servers designated as the primary instance or the corresponding component of each of the other of the plurality of transaction processing servers designated as the backup instance is operating correctly. 
     
     
       3. The system of  claim 1 , wherein only the one of the plurality of transaction processing servers designated as the primary instance is configured to transmit a data message indicative of the generated result of the attempt to satisfy one or both of the incoming request and the previously received request to a recipient external to the system. 
     
     
       4. The system of  claim 1 , wherein each of the plurality of fault tolerant processors periodically receives a communication from others of the plurality of fault tolerant processors, wherein a failure to receive the communication from one of the plurality of fault tolerant processors is deemed to indicate a failure of the transaction processing server coupled therewith. 
     
     
       5. The system of  claim 1 , wherein each of the plurality of components of each of the plurality of transaction processing servers are further coupled with the one of the plurality of fault tolerance processors coupled with that transaction processing server such that the one of the plurality of fault tolerance processors is able to detect a failure of any of the plurality of components. 
     
     
       6. The system of  claim 1 , wherein the current state of each of the plurality of components of each of the other of the plurality of transaction processing servers designated as the backup instance corresponds to a state of each of the plurality of components of the one of the plurality of transaction processing servers designated as the primary instance subsequent to the processing of an electronic transaction message preceding the electronic transaction message currently being processed by the one of the plurality of transaction processing servers designated as the primary instance. 
     
     
       7. The system of  claim 1 , wherein the plurality of fault tolerance processors, establish, upon the plurality of transaction processing servers starting up, which of the plurality of transaction processing servers is designated as the primary instance. 
     
     
       8. The system of  claim 1 ,
 wherein each of the other of the plurality of transaction processing servers designated as the backup instance includes a queue configured to store previously received electronic messages from the one of the plurality of transaction processing servers designated as the primary instance, and 
 wherein the one of the plurality of transaction processing servers which is designated as the new primary instance is configured to process all of the previously received electronic messages stored in the queue before becoming the new primary instance. 
 
     
     
       9. The system of  claim 8 ,
 wherein the electronic message indicative of the result of the operation of the one of the plurality of transaction processing servers designated as the primary instance includes a primary result output, and 
 wherein the electronic message indicative of the result of the operation of each of the other of the plurality of transaction processing servers designated as the backup instance includes a backup result output. 
 
     
     
       10. The system of  claim 9 , wherein at least a subset of the plurality of components of the one of other of the plurality of transaction processing servers designated as the new primary instance are configured to complete a reconciliation of the backup result output with the primary result output. 
     
     
       11. The system of  claim 9 , wherein at least a subset of the plurality of components of the one of other of the plurality of transaction processing servers designated as the new primary instance are configured to identify a backup result output that does not have a respective primary result output and publish an unmatched result output via a shared memory architecture. 
     
     
       12. The system of  claim 1 , wherein when the one of the plurality of fault tolerance processors coupled with the primary instance determines that the failure has occurred in the primary instance, each of the plurality of fault tolerance processors is configured to cause the primary instance to at least de-couple from the other of the plurality of transaction processing servers. 
     
     
       13. The system of  claim 1 ,
 wherein a first component of the plurality of components comprises a transaction receiver processor for receiving the electronic message via the electronic communications network, the electronic message including data indicative of an order to trade, 
 wherein a second component of the plurality of components comprises a transaction processor for matching the received order to trade with one or more of previously received unmatched orders which are counter thereto, and 
 wherein a third component of the plurality of components comprises a result generator processor for generating and publishing a market data message. 
 
     
     
       14. A computer implemented method comprising:
 providing a plurality of transaction processing servers coupled with each other via an electronic communications network each of the plurality of transaction processing servers comprising a processor; 
 implementing, by the processor of each of the plurality of transaction processing servers, a plurality of components coupled with each other, each of the plurality of components characterized by a current state and configured to perform one or more of receiving an electronic message, processing the received electronic message and generating an electronic result message indicative of a result of an operation which may alter the current state thereof, wherein one of the plurality of transaction processing servers is designated as a primary instance and each other of the plurality of transaction processing servers is designated as a backup instance, wherein each of the plurality of transaction processing servers is operative to receive an incoming request for a transaction, process the received incoming request to identify a previously received request for a transaction counter thereto in an attempt to satisfy one or both of the incoming request and the previously received request and generate a result indicative thereof, the transaction processing server designated as the primary instance at least commencing processing of the incoming request prior to each of the other of the plurality of transaction processing servers designated as the backup instance, wherein one or more of the plurality of components of the transaction processing server designated as the primary instance are each configured to communicate the electronic result message indicative of the result of the operation thereof to a corresponding component of each of the other of the plurality of transaction processing servers designated as the backup instance; 
 determining, by a plurality of fault tolerance processors, each of the plurality of fault tolerance processors coupled with one transaction processing server of the plurality of transaction processing servers, that a failure has occurred in the transaction processing server designated as the primary instance; 
 causing, by the plurality of fault tolerance processors, one of the other of the plurality of transaction processing servers to be designated as a new primary instance; 
 synchronizing, by the transaction processing server which is designated as the new primary instance, the current state of each of the plurality of components thereof based on received electronic messages indicative of the result of the operation of the corresponding component of the primary instance; and 
 processing, by the transaction processing server, any incoming requests not processed by the primary instance before commencing processing of new incoming requests. 
 
     
     
       15. The method of  claim 14 , wherein upon receipt of the electronic result message indicative of the result of the operation of the one or more of the plurality of components of the primary instance, the corresponding component of each of the other of the plurality of transaction processing servers designated as the backup instance compares a result of the operation thereof with the result indicated by the received electronic result message to determine whether one of the plurality of components of the one of the plurality of transaction processing servers designated as the primary instance or the corresponding component of the other of the plurality of transaction processing servers designated as the backup instance is operating correctly. 
     
     
       16. The method of  claim 14 , further comprising:
 transmitting, by only the one of the plurality of transaction processing servers designated as the primary instance, a data message indicative of the generated result of the attempt to satisfy one or both of the incoming request and the previously received request to an external recipient. 
 
     
     
       17. The method of  claim 14 , further comprising:
 receiving periodically, by each of the plurality of fault tolerant processors, a communication from others of the plurality of fault tolerant processors, wherein a failure to receive the communication from one of the plurality of fault tolerant processors is deemed to indicate a failure of the transaction processing server coupled therewith. 
 
     
     
       18. The method of  claim 14 , wherein each of the plurality of components of each of the plurality of transaction processing servers are further coupled with the one of the plurality of fault tolerance processors coupled with that transaction processing server such that the one of the plurality of fault tolerance processors is able to detect a failure of any of the plurality of components. 
     
     
       19. The method of  claim 14 , wherein the current state of each of the plurality of components of each of the other of the plurality of transaction processing servers designated as the backup instance corresponds to a state of each of the plurality of components of the one of the plurality of transaction processing servers designated as the primary instance subsequent to the processing of an electronic transaction message preceding the electronic transaction message currently being processed by the one of the plurality of transaction processing servers designated as the primary instance. 
     
     
       20. The method of  claim 14 , further comprising:
 establishing, by the plurality of fault tolerance processors, upon the plurality of transaction processing servers starting up, which of the plurality of transaction processing servers is designated as the primary instance. 
 
     
     
       21. The method of  claim 14 ,
 wherein each of the other of the plurality of transaction processing servers designated as the backup instance includes a queue configured to store previously received electronic messages from the one of the plurality of transaction processing servers designated as the primary instance, and 
 wherein the one of the other of the plurality of transaction processing servers which is designated as the new primary instance is configured to process all of the previously received electronic messages stored in the queue before becoming the new primary instance. 
 
     
     
       22. The method of  claim 21 ,
 wherein the electronic message indicative of the result of the operation of the one of the plurality of transaction processing servers designated as the primary instance includes a primary result output, and 
 wherein the electronic message indicative of the result of the operation of each of the other of the plurality of transaction processing servers designated as the backup instance includes a backup result output. 
 
     
     
       23. The method of  claim 22 , wherein at least a subset of the plurality of components of the one of other of the plurality of transaction processing servers designated as the new primary instance are configured to complete a reconciliation of the backup result output with the primary result output. 
     
     
       24. The method of  claim 21 , wherein at least a subset of the plurality of components of the one of the other of the plurality of transaction processing servers designated as the new primary instance are configured to identify a backup result output that does not have a respective primary result output and publish an unmatched result output via a shared memory architecture. 
     
     
       25. The method of  claim 14 , further comprising:
 when the one of the plurality of fault tolerance processors coupled with the primary instance determines that the failure has occurred in the primary instance, causing, by each of the plurality of fault tolerance processors, the primary instance to at least de-couple from the other of the plurality of transaction processing servers. 
 
     
     
       26. The method of  claim 14 ,
 wherein a first component of the plurality of components comprises a transaction receiver processor for receiving the electronic message via the electronic communications network, the electronic message including data indicative of an order to trade, 
 wherein a second component of the plurality of components comprises a transaction processor for matching the received order to trade with one or more of previously received unmatched orders which are counter thereto, and 
 wherein a third component of the plurality of components comprises a result generator processor for generating and publishing a market data message. 
 
     
     
       27. A system comprising:
 means for providing a plurality of transaction processing servers coupled with each other via an electronic communications network; 
 means for implementing a plurality of components of each of the plurality of transaction processing servers, coupled with each other, each of the plurality of components characterized by a current state and configured to perform one or more of receiving an electronic message, processing the received electronic message and generating an electronic result message indicative of a result of an operation which may alter the current state thereof, wherein one of the plurality of transaction processing servers is designated as a primary instance and each other of the plurality of transaction processing servers is designated as a backup instance, wherein each of the plurality of transaction processing servers is operative to receive an incoming request for a transaction, process the received incoming request to identify a previously received request for a transaction counter thereto in an attempt to satisfy one or both of the incoming request and the previously received request and generate a result indicative thereof, the transaction processing server designated as the primary instance at least commencing processing of the incoming request prior to each of the other of the plurality of transaction processing servers designated as the backup instance, wherein one or more of the plurality of components of the transaction processing server designated as the primary instance are each configured to communicate the electronic result message indicative of the result of the operation thereof to a corresponding component of each of the other of the plurality of transaction processing servers designated as the backup instance; 
 means for determining that a failure has occurred in the transaction processing server designated as the primary instance; 
 means for causing one of the other of the plurality of transaction processing servers to be designated as a new primary instance; 
 means for synchronizing the current state of each of the plurality of components thereof based on received electronic messages indicative of the result of the operation of the corresponding component of the primary instance; and 
 means for processing any incoming requests not processed by the primary instance before commencing processing of new incoming requests.