Automatic failover configuration with lightweight observer

Abstract

Techniques used in an automatic failover configuration having a primary database system, a standby database system, and an observer for preventing divergence among the primary and standby database systems while increasing the availability of the primary database system. In the automatic failover configuration, the primary database system remains available even in the absence of both the standby and the observer as long as the standby and the observer become absent sequentially. The failover configuration further permits automatic failover only when the observer is present and the standby and the primary are synchronized and inhibits state changes during failover. The database systems and the observer have copies of failover configuration state and the techniques include techniques for propagating the most recent version of the state among the databases and the observer and techniques for using carefully-ordered writes to ensure that state changes are propagated in a fashion which prevents divergence.

1. An automatic failover configuration comprising:
 a primary database system on a first host machine operating in a first database server that processes transactions and produces redo data therefor as a primary database system participant; 
 a standby database system on a second host machine operating in a second database server that receives the redo data via a redo communications link as a standby database system participant; and 
 an active observer, which is a client of the first and second database server, that provides a quorum for a failover operation in which the standby database system participant becomes the primary database system participant, the active observer exchanging first control messages with the primary database system and the standby database system via one or more non-redo communications links, 
 the primary database system and the standby database system exchanging second control messages via the one or more non-redo communications links; 
 the active observer being an independently executing entity from the primary database system and the standby database system, the active observer executing on system which is coupled to the non-redo communications links, and the active observer employing the same interface to communicate with the primary database system and the standby database system as any other client of the database servers; and 
 wherein the first and second control messages propagate a current automatic failover configuration state among participants of the automatic failover configuration, the current automatic failover configuration state including an indication which changes when the active observer is to request further state information from the primary database system, the active observer responding to the changed indication by requesting the further state information from the primary database system. 
 
     
     
       2. The automatic failover configuration set forth in  claim 1  wherein:
 the further state information includes an identifier for the active observer; and when an observer starts up, the observer that is starting up requests the further state information, the primary database system responding to the request with the identifier for the active observer only if there is currently no active observer in the configuration; and 
 the observer terminates if it does not receive the identifier for the active observer. 
 
     
     
       3. The automatic failover configuration set forth in  claim 1  wherein:
 the further state information includes an identifier for the active observer; and when the observer identifier returned to the active observer in a response to a request for further state information is different from the active observer's observer identifier, the active observer terminates. 
 
     
     
       4. The automatic failover configuration set forth in  claim 1  wherein: the client of the first and second database servers has a hardware and/or operating system platform that is different from the hardware and/or operating system platform used in the primary and standby database systems. 
     
     
       5. Data storage apparatus characterized in that: the data storage device contains code which, when executed, implements an automatic failover configuration, comprising:
 a primary database system operating in a first database server that processes transactions and produces redo data therefor as a primary database system participant; 
 a standby database system operating in a second database server that receives the redo data via a redo communications link as a standby database system participant; and 
 an active observer that provides a quorum for a failover operation in which the standby database system participant becomes the primary database system participant, 
 the active observer exchanging first control messages with the primary database system and the standby database system via one or more non-redo communications links, 
 the primary database system and the standby database system exchanging second control messages via the one or more non-redo communications links; 
 the active observer being an independently executing entity from the primary database system and the standby database system, the active observer executing on a system which is coupled to the non-redo communications links, and the active observer employing the same interface to communicate with the primary database system and the standby database system as any other client of the database servers; and 
 wherein the first and second control messages propagate a current automatic failover configuration state among participants of the automatic failover configuration, the current automatic failover configuration state including an indication which changes when the active observer is to request further state information from the primary database system, the active observer responding to the changed indication by requesting the further state information from the primary database system. 
 
     
     
       6. A method practiced in an automatic failover configuration which comprises a primary database system on a first host machine as a primary database system participant, a standby database system on a second host machine as a standby database system participant, an active observer having an active observer identifier and a communications link for communicating automatic failover configuration state among participants of the automatic failover configuration, the active observer being an independently executing entity from the primary database system and the standby database system,
 the method preventing divergence of the database systems resulting from an automatic failover and comprising the steps performed in the active observer of: sending a first message to the standby database system indicating that the active observer has determined that a failover condition has occurred; 
 receiving a second message from the standby database system indicating that the standby database system has entered a failover pending state indicating that the automatic failover configuration is ready to failover; 
 responding thereto by entering the failover pending state; 
 responding to a third message from the standby database system indicating that the standby database system has completed the failover and is currently the primary database system by leaving the failover pending state, the active observer performing the steps before leaving the failover pending state of: 
 requesting a valid active observer identifier from the primary database system; 
 
       if no valid active observer identifier is received, terminating; and
 if a valid active observer identifier is received, making the received valid active observer identifier the active observer identifier; 
 receiving current automatic failover configuration state from the current primary database system, wherein the current automatic failover configuration state includes an indication from which the active observer can determine whether there is another active observer; 
 determining from the indication whether there is another active observer; and 
 
       terminating if there is another active observer. 
     
     
       7. The method set forth in  claim 6  wherein:
 the current automatic failover configuration state includes a current active observer identifier, and the active observer performs the steps of: 
 comparing the current active observer identifier with the active observer's active observer identifier; and 
 if the current active observer identifier and the active observer's active observer identifier are different, terminating. 
 
     
     
       8. Data storage apparatus characterized in that: the data storage device contains code which, when executed implements a method practiced in an automatic failover configuration which comprises a primary database system as a primary database system participant, a standby database system as a standby database system participant, an active observer having an active observer identifier and a communications link for communicating automatic failover configuration state among participants of the automatic failover configuration, the active observer being an independently executing entity from the primary database system and the standby database system,
 the method preventing divergence of the database systems resulting from an automatic failover and comprising the steps performed in the active observer of: 
 sending a first message from the active observer to the standby database system indicating that the active observer has determined that a failover condition has occurred; 
 receiving a second message by the active observer from the standby database system indicating that the standby database system has entered a failover pending state indicating that the automatic failover configuration is ready to failover; 
 responding thereto by the active observer by entering the failover pending state; and 
 responding to a third message by the active observer from the standby database system indicating that the standby database system has completed the failover and is currently the primary database system by leaving the failover pending state, the active observer performing the steps before leaving the failover pending state of: 
 requesting a valid active observer identifier from the primary database system; 
 if no valid active observer identifier is received, terminating; 
 if a valid active observer identifier is received, making the received valid active observer identifier the active observer identifier; and 
 receiving current automatic failover configuration state from the current primary database system, wherein the current automatic failover configuration state includes an indication from which the active observer can determine whether there is another active observer; 
 determining from the indication whether there is another active observer; and 
 
       terminating if there is another active observer. 
     
     
       9. A method practiced in an automatic failover configuration which comprises a primary database system on a first host machine as a primary database system participant, a standby database system on a second host machine as a standby database system participant, and an active observer and a communications link for communicating automatic failover configuration state among participants of the automatic failover configuration, the active observer being an independently executing entity from the primary database system and the standby database system, the active observer having a unique active observer identifier and the automatic failover configuration state including the current active observer identifier and the method ensuring that there is only one active observer in the automatic failover configuration and comprising the steps performed in an observer of:
 on starting up,
 requesting an active observer identifier from the primary database system, and 
 if no active observer identifier is received, terminating and on receiving automatic failover configuration state, 
 when the observer's current automatic failover configuration state indicates that an automatic failover is occurring, performing the steps prior to altering the current automatic failover configuration state to indicate that no automatic failover is occurring of: 
 requesting an active observer identifier from the primary database system, and terminating if no active observer identifier is received; 
 comparing the observer's active observer identifier with the current active observer identifier, and 
 if the observer's active observer identifier is different from the current active observer identifier, terminating. 
 
 
     
     
       10. Data storage apparatus characterized in that: the data storage device contains code which, when executed implements a method practiced in an automatic failover configuration which comprises a primary database system as primary database system participant, a standby database system as a standby database system participant, and an active observer and a communications link for communicating automatic failover configuration state among participants of the automatic failover configuration, the active observer being an independently executing entity from the primary database system and the standby database system, the active observer having a unique active observer identifier and the automatic failover configuration state including the current active observer identifier and the method ensuring that there is only one active observer in the automatic failover configuration and comprising the steps performed in an observer of:
 on starting up,
 requesting an active observer identifier from the primary database system, and 
 if no active observer identifier is received, terminating and on receiving automatic failover configuration state, 
 when the observer's current automatic failover configuration state indicates that an automatic failover is occurring, performing the steps prior to altering the current automatic failover configuration state to indicate that no automatic failover is occurring of: 
 requesting an active observer identifier from the primary database system, and terminating if no active observer identifier is received; 
 comparing the observer's active observer identifier with the current active observer identifier, and 
 if the observer's active observer identifier is different from the current active observer identifier, terminating.