Remote checkpoint memory system and protocol for fault-tolerant computer system

Abstract

A mechanism for maintaining a consistent, periodically updated state in main memory without constraining normal computer operation is provided, thereby enabling a computer system to recover from faults without loss of data or processing continuity. In this invention, a first computer includes a processor and input/output elements connected to a main memory subsystem including a primary element. A second computer has a remote checkpoint memory element, which may include one or more buffer memories and a shadow memory, which is connected to the main memory subsystem of the first computer. During normal processing, an image of data written to the primary memory element is captured by the remote checkpoint memory element. When a new checkpoint is desired (thereby establishing a consistent state in main memory to which all executing applications can safely return following a fault), the data previously captured is used to establish a new checkpointed state in the second computer. In case of failure of the first computer, the second computer can be restarted to operate from the last checkpoint established for the first computer. This structure and protocol can guarantee a consistent state in main memory, thus enabling fault-tolerant operation.

What is claimed is: 
     
       1. A computer system comprising: a first computer processing at least a first application, the first computer including: a processor having a cache, internal registers and an input/output event queue,   a main memory subsystem coupled to the processor and cache,   a write buffer coupled to the main memory subsystem that captures data written to the main memory,   an external port, and   an interface controller, coupled to the write buffer and the external port, that transfers data in the write buffer to the external port of the first computer,   wherein the processor includes means for flushing the cache, internal registers and input/output event queue to the main memory subsystem and means for issuing a checkpoint instruction after flushing the cache, internal registers and input/output event queue;     a data communications link coupled to the external port of the first computer;   a second computer including: an external port coupled to the data communications link,   an interface controller, coupled to the external port of the second computer to receive the data from the interface controller of the first computer,   a main memory subsystem,   a buffer memory, coupled to the interface controller of the second computer and the main memory of the second computer, that receives the data transferred from the write buffer of the first computer; and   wherein data stored in the buffer memory is transferred to the main memory subsystem of the second computer upon receipt of the checkpoint instruction from the first computer, such that the main memory subsystem of the second computer maintains a consistent state from which processing of the first application can be restarted of the first computer.     
     
     
       2. The computer system of claim 1, wherein the second computer further includes: a processor coupled to the main memory subsystem; and wherein, upon a failure of the first computer, the second computer continues processing the first application of the first computer without data loss.   
     
     
       3. The computer system of claim 2, wherein the second computer processes at least a second application, and wherein processing of the second application is terminated when a failure occurs in the first computer. 
     
     
       4. The computer system of claim 2, wherein the second computer processes at least a second application, and wherein the second computer processes both the first and the second application when a failure occurs in the first computer. 
     
     
       5. The computer system of claim 2, wherein the second computer further includes: a write buffer coupled to the main memory subsystem that captures data written to the main memory subsystem of the second computer; and   wherein the interface controller of the second computer transfers data contained in the write buffer of the second computer to the first computer, so that the main memory subsystem of the first computer acts as the shadow memory for the main memory of the second computer.   
     
     
       6. A computer system comprising: at least three computers, each of the computers except one spare computer performing data processing tasks;   a data communications network coupled to each of the computers such that the plurality of computers are connected in a logical ring;   wherein each computer includes a processor having a cache, internal registers and input/output event queue and connected to a main memory and a write buffer coupled to capture data written to the memory by the processor;   means for flushing the cache, internal registers and input/output event queue to the main memory and means for copying data from the write buffer to the main memory of an adjacent computer in the logical ring at periodic checkpoints; and   wherein upon failure of one of the computers performing data processing tasks, the data processing tasks of the computers performing data processing tasks are performed by functional computers of the computers including the spare computer without loss of data.   
     
     
       7. The computer system of claim 6, further comprising a plurality of dual ported I/O devices, each of the dual ported I/O devices being coupled to at least two of the plurality of computers. 
     
     
       8. The computer system of claim 6, wherein a number of computers performing data processing tasks are between the spare computer and the failed computer, and wherein the data processing tasks of the failed computer and each of the number of computers are performed by the next adjacent computer such that the spare computer performs the tasks of one of the number of computers. 
     
     
       9. A method for providing fault tolerance in a computer system having first and second computers, each of the first and second computers having a processor with a cache, internal registers and an input/output event queue an external port, a main memory and a buffer memory, the method including steps of: capturing data written to the main memory of the first computer;   transferring the data over a data link to the buffer memory of the second computer;   flushing the cache, internal registers and input/output event queue of the processor of the first computer such that data contained within the cache, internal registers and input/output event queue is written to the main memory of the first computer and to the buffer memory of the second computer;   copying data from the buffer memory of the second computer to the main memory of the second computer such that a checkpoint is established and the main memory of the second computer acts as a shadow memory of the main memory of the first computer.   
     
     
       10. The method of claim 9, further comprising steps of: processing a first application in the first computer; and   processing the first application in the second computer upon failure of the first computer.   
     
     
       11. The method of claim 10 further comprising steps of: processing a second application in the second computer; and   terminating the processing of the second application upon failure of the first computer.   
     
     
       12. The method of claim 10 further comprising steps of: processing a second application in the second computer; and   processing both the first and the second applications in the second computer upon failure of the first computer.   
     
     
       13. The method of claim 9, further comprising steps of: capturing data written to the main memory of the second computer;   transferring the data over a data link to the buffer memory of the first computer;   flushing the cache of the processor of the second computer such that data contained within the cache is written to the main memory of the second computer and the buffer memory of the first computer;   copying data from the buffer memory of the first computer to the main memory of the first computer such that a checkpoint is established and the main memory of the first computer acts as a shadow memory of the main memory of the second computer.   
     
     
       14. A computer system comprising: a processor;   a main memory subsystem, coupled to the processor, including: a primary memory element from which data is read and to which data is written by the processor;   a write buffer that monitors each time data is written to the primary memory element by the processor and stores buffer data related to the data written to the processor;   wherein the processor and the primary memory element are in a first computer and wherein the write buffer is in a second computer having a main memory subsystem and connected to the first computer by a communication link; means, using the buffer data, for ensuring the existence of a consistent checkpoint state in the main memory subsystem of the second computer to which processing can resume without loss of data integrity or program continuity following a fault; and   wherein the processor has a corresponding input/output subsystem which provides input/output events initiated by the processor, wherein the processor has means for queuing input/output events between checkpoints and means for flushing the queued events to the primary memory element when a checkpoint is to be established, whereby input/output events are captured in checkpoint data in the main memory subsystem of the second computer.