Apparatus and method for quorum-based power-down of unresponsive servers in a computer cluster

Abstract

An apparatus and method provide a quorum-based server power-down mechanism that allows a manager in a computer cluster to power-down unresponsive servers in a manner that assures that an unresponsive server does not become responsive again. In order for a manager in a cluster to power down servers in the cluster, the cluster must have quorum, meaning that a majority of the computers in the cluster must be responsive. If the cluster has quorum, and if the manager server did not fail, the manager causes the failed server(s) to be powered down. If the manager server did fail, the new manager causes all unresponsive servers in the cluster to be powered down. If the power-down is successful, the resources on the failed server(s) may be failed over to other servers in the cluster that were not powered down. If the power-down is not successful, the cluster is disabled.

1 . An apparatus comprising: 
 at least one processor;    a memory coupled to the at least one processor;    a server process residing in the memory and executed by the at least one processor;    a cluster engine residing in the memory and executed by the at least one processor, the cluster engine handling communications between the server process and other servers in a cluster; and    a quorum-based server power-down mechanism residing in the memory and executed by the at least one processor, the quorum-based server power-down mechanism determining whether the server process is part of a group of servers that include a majority of servers in the cluster, and if so, the quorum-based server power-down mechanism issuing a command to power down at least one of the other servers in the cluster.    
   
   
       2 . The apparatus of  claim 1  further comprising a service processor that receives the command, and in response, powers down the at least one of the other servers.  
   
   
       3 . The apparatus of  claim 1  wherein the cluster engine comprises a heartbeat mechanism that sends a periodic message to the other servers in the cluster to indicate the server process is functioning properly.  
   
   
       4 . The apparatus of  claim 3  wherein the heartbeat mechanism receives periodic messages from the other servers in the cluster that indicate the other servers in the cluster are functioning properly.  
   
   
       5 . The apparatus of  claim 4  wherein the cluster engine comprises a membership change mechanism that generates a membership change message to all servers in the cluster when a selected server in the cluster becomes unresponsive by failing to send a periodic message that indicates the selected server is functioning properly.  
   
   
       6 . The apparatus of  claim 1  wherein the cluster engine comprises a membership change mechanism that generates a membership change message to all servers in the cluster when any of the servers in the cluster become unresponsive.  
   
   
       7 . The apparatus of  claim 1  wherein the quorum-based server power-down mechanism determines whether a manager of the cluster failed, and if so, the quorum-based server power-down mechanism issues at least one command to power down all unresponsive servers in the cluster.  
   
   
       8 . The apparatus of  claim 1  wherein the quorum-based server power-down mechanism determines whether a manager of the cluster failed when an indication of a server failure is received, and if not, the quorum-based server power-down mechanism issues at least one command to power down a server corresponding to the received indication of server failure.  
   
   
       9 . The apparatus of  claim 1  wherein the server process resides in a logical partition defined on the apparatus.  
   
   
       10 . The apparatus of  claim 1  wherein the quorum-based server power-down mechanism determines whether the power down of the at least one of the other servers was successful, and if the power down of the at least one of the other servers was successful, the quorum-based server power-down mechanism enables failing over any resources on the at least one of the other servers that was powered down to at least one server that is responsive, and if the power down of the at least one of the other servers was not successful, the quorum-based server power-down mechanism disables the cluster.  
   
   
       11 . A networked computer system comprising: 
 a plurality of servers coupled together via a network into a cluster, each server comprising: 
 a cluster engine that handles communications between servers in the cluster; and  
 a quorum-based server power-down mechanism that determines whether a server is part of a group of servers that includes a majority of servers in the cluster, and if so, the quorum-based server power-down mechanism issuing a command to power down at least one server in the cluster.  
   
   
   
       12 . The networked computer system of  claim 11  further comprising a service processor coupled to each of the plurality of servers in the cluster, the service processor receiving the command, and in response, powering down the at least one server in the cluster.  
   
   
       13 . The networked computer system of  claim 11  wherein the cluster engine comprises a heartbeat mechanism that sends a periodic message to other servers in the cluster to indicate the server is functioning properly.  
   
   
       14 . The networked computer system of  claim 13  wherein the heartbeat mechanism receives periodic messages from the other servers in the cluster that indicate the other servers in the cluster are functioning properly.  
   
   
       15 . The networked computer system of  claim 14  wherein the cluster engine comprises a membership change mechanism that generates a membership change message to all servers in the cluster when a selected server in the cluster becomes unresponsive by failing to send a periodic message that indicates the selected server is functioning properly.  
   
   
       16 . The networked computer system of  claim 11  wherein the cluster engine comprises a membership change mechanism that generates a membership change message to all servers in the cluster when any of the servers in the cluster become unresponsive.  
   
   
       17 . The networked computer system of  claim 11  wherein the quorum-based server power-down mechanism determines whether a manager of the cluster failed, and if so, the quorum-based server power-down mechanism issues at least one command to power down all unresponsive servers in the cluster.  
   
   
       18 . The networked computer system of  claim 11  wherein the quorum-based server power-down mechanism determines whether a manager of the cluster failed when an indication of a server failure is received, and if not, the quorum-based server power-down mechanism issues at least one command to power down a server corresponding to the received indication of server failure.  
   
   
       19 . The networked computer system of  claim 11  wherein at least one server in the cluster resides in a logical partition.  
   
   
       20 . The networked computer system of  claim 11  wherein the quorum-based server power-down mechanism determines whether the power down of the at least one of the other servers was successful, and if the power down of the at least one of the other servers was successful, the quorum-based server power-down mechanism enables failing over any resources on the at least one of the other servers that was powered down to at least one server that is responsive, and if the power down of the at least one of the other servers was not successful, the quorum-based server power-down mechanism disables the cluster.  
   
   
       21 . An apparatus comprising: 
 (A) at least one processor;    (B) a memory coupled to the at least one processor;    (C) a server process residing in the memory and executed by the at least one processor;    (D) a cluster engine residing in the memory and executed by the at least one processor, the cluster engine handling communications between the server process and other servers in a cluster, the cluster engine comprising: 
 (D1) a heartbeat mechanism that sends a periodic message to the other servers in the cluster to indicate the server process is functioning properly and that receives periodic messages from the other servers in the cluster that indicate the other servers in the cluster are functioning properly;  
 (D2) a membership change mechanism that generates a membership change message to all servers in the cluster when any of the servers in the cluster become unresponsive;  
   (E) a quorum-based server power-down mechanism residing in the memory and executed by the at least one processor, the quorum-based server power-down mechanism determining whether the server process is part of a group of servers that includes a majority of servers in the cluster, and if so, the quorum-based server power-down mechanism determining whether a manager of the cluster failed when an indication of a server failure is received, and if a manager of the cluster failed, the quorum-based server power-down mechanism issues at least one command to power down all unresponsive servers in the cluster, and if a manager of the cluster did not fail, the quorum-based server power-down mechanism issues at least one command to power down a server corresponding to the received indication of server failure.    
   
   
       22 . The apparatus of  claim 21  further comprising a service processor that receives the command, and in response, powers down at least one of the other servers.  
   
   
       23 . The apparatus of  claim 21  wherein the server process resides in a logical partition defined on the apparatus.  
   
   
       24 . The apparatus of  claim 21  wherein the quorum-based server power-down mechanism determines whether the power down of the at least one of the other servers was successful, and if the power down of the at least one of the other servers was successful, the quorum-based server power-down mechanism enables failing over any resources on the at least one of the other servers that was powered down to at least one server that is responsive, and if the power down of the at least one of the other servers was successful, the quorum-based server power-down mechanism disables the cluster.  
   
   
       25 . A computer-implemented method for handling an unresponsive server in a cluster, the method comprising the steps of: 
 determining when a server in the cluster becomes unresponsive;    determining whether a majority of servers in the cluster are responsive; and    if a majority of servers in the cluster are responsive, issuing a command to power down at least one server in the cluster.    
   
   
       26 . The method of  claim 25  further comprising the step of each server in the cluster sending a periodic message to other servers in the cluster to indicate proper function of the server sending the periodic message.  
   
   
       27 . The method of  claim 25  further comprising the step of each server receiving periodic messages from other servers in the cluster that indicate the other servers in the cluster are functioning properly.  
   
   
       28 . The method of  claim 27  further comprising the step of generating a membership change message to all servers in the cluster when a selected server in the cluster becomes unresponsive by failing to send a periodic message that indicates the selected server is functioning properly.  
   
   
       29 . The method of  claim 25  further comprising the step of generating a membership change message to all servers in the cluster when any of the servers in the cluster become unresponsive.  
   
   
       30 . The method of  claim 25  further comprising the step of determining whether a manager of the cluster failed, and if so, issuing at least one command to power down all unresponsive servers in the cluster.  
   
   
       31 . The method of  claim 25  further comprising the step of determining whether a manager of the cluster failed when an indication of a server failure is received, and if not, issuing at least one command to power down a server corresponding to the received indication of server failure.  
   
   
       32 . The method of  claim 25  further comprising the steps of: 
 determining whether the power down of the at least one of the other servers was successful;    if the power down of the at least one of the other servers was successful, enabling the failing over any resources on the at least one of the other servers that was powered down to at least one server that is responsive; and    if the power down of the at least one of the other servers was not successful, disabling the cluster.    
   
   
       33 . A computer-implemented method for handling an unresponsive server in a cluster, the method comprising the steps of: 
 each server in the cluster sending a periodic message to other servers in the cluster to indicate proper function of the server sending the periodic message;    each server in the cluster receiving periodic messages from other servers in the cluster that indicate the other servers in the cluster are functioning properly;    generating a membership change message to all servers in the cluster when any of the servers in the cluster become unresponsive;    determining whether a majority of servers in the cluster are responsive;    receiving an indication of a server failure;    if the majority of servers in the cluster are responsive, performing the steps of: 
 determining whether the indication of the server failure indicates a manager of the cluster failed;  
 if the manager of the cluster failed, issuing at least one command to power down all unresponsive servers in the cluster; and  
 if the manager of the cluster did not fail, issuing at least one command to power down a server corresponding to the received indication of server failure.  
   
   
   
       34 . The method of  claim 33  further comprising the steps of: 
 determining whether the power down of the at least one of the other servers was successful;    if the power down of the at least one of the other servers was successful, enabling the failing over any resources on the at least one of the other servers that was powered down to at least one server that is responsive; and    if the power down of the at least one of the other servers was not successful, disabling the cluster.    
   
   
       35 . A program product comprising: 
 a quorum-based server power-down mechanism that determines whether a server is part of a group of servers that include a majority of servers in a cluster, and if so, the quorum-based server power-down mechanism issues a command to power down at least one server in the cluster; and    computer readable signal bearing media bearing the quorum-based server power-down mechanism.    
   
   
       36 . The program product of  claim 35  wherein the signal bearing media comprises recordable media.  
   
   
       37 . The program product of  claim 35  wherein the signal bearing media comprises transmission media.  
   
   
       38 . The program product of  claim 35  further comprising a heartbeat mechanism that sends a periodic message to the other servers in the cluster to indicate the server process is functioning properly.  
   
   
       39 . The program product of  claim 38  wherein the heartbeat mechanism receives periodic messages from the other servers in the cluster that indicate the other servers in the cluster are functioning properly.  
   
   
       40 . The program product of  claim 39  further comprising a membership change mechanism that generates a membership change message to all servers in the cluster when a selected server in the cluster becomes unresponsive by failing to send a periodic message that indicates the selected server is functioning properly.  
   
   
       41 . The program product of  claim 35  further comprising a membership change mechanism that generates a membership change message to all servers in the cluster when any of the servers in the cluster become unresponsive.  
   
   
       42 . The program product of  claim 35  wherein the quorum-based server power-down mechanism determines whether a manager of the cluster failed, and if so, the quorum-based server power-down mechanism issues at least one command to power down all unresponsive servers in the cluster.  
   
   
       43 . The program product of  claim 35  wherein the quorum-based server power-down mechanism determines whether a manager of the cluster failed when an indication of a server failure is received, and if not, the quorum-based server power-down mechanism issues at least one command to power down a server corresponding to the received indication of server failure.  
   
   
       44 . The program product of  claim 35  wherein at least one server in the cluster resides in a logical partition.  
   
   
       45 . The program product of  claim 35  wherein the quorum-based server power-down mechanism determines whether the power down of the at least one of the other servers was successful, and if the power down of the at least one of the other servers was successful, the quorum-based server power-down mechanism enables failing over any resources on the at least one of the other servers that was powered down to at least one server that is responsive, and if the power down of the at least one of the other servers was not successful, the quorum-based server power-down mechanism disables the cluster.  
   
   
       46 . A program product comprising: 
 (A) a cluster engine residing in the memory and executed by the at least one processor, the cluster engine handling communications between a plurality of servers in a cluster, the cluster engine comprising: 
 (A1) a heartbeat mechanism that sends a periodic message to the other servers in the cluster to indicate the server process is functioning properly and that receives periodic messages from the other servers in the cluster that indicate the other servers in the cluster are functioning properly;  
 (A2) a membership change mechanism that generates a membership change message to all servers in the cluster when any of the servers in the cluster become unresponsive; and  
 (A3) a quorum-based server power-down mechanism that determines whether the server process is part of a group of servers that includes a majority of servers in the cluster, and if so, the quorum-based server power-down mechanism determines whether a manager of the cluster failed when an indication of a server failure is received, and if a manager of the cluster failed, the quorum-based server power-down mechanism issues at least one command to power down all unresponsive servers in the cluster, and if a manager of the cluster did not fail, the quorum-based server power-down mechanism issues at least one command to power down a server corresponding to the received indication of server failure; and  
   (B) computer readable signal bearing media bearing the cluster engine.    
   
   
       47 . The program product of  claim 46  wherein the signal bearing media comprises recordable media.  
   
   
       48 . The program product of  claim 46  wherein the signal bearing media comprises transmission media.  
   
   
       49 . The program product of  claim 46  wherein at least one server in the cluster resides in a logical partition.  
   
   
       50 . The program product of  claim 46  wherein the quorum-based server power-down mechanism determines whether the power down of the at least one of the other servers was successful, and if the power down of the at least one of the other servers was successful, the quorum-based server power-down mechanism enables failing over any resources on the at least one of the other servers that was powered down to at least one server that is responsive, and if the power down of the at least one of the other servers was not successful, the quorum-based server power-down mechanism disables the cluster.