Distributed management of a storage system

Abstract

Systems, apparatuses, methods, and computer-readable storage mediums for performing lease-based fencing using a time-limited lease window. During the time-limited lease window, writes to a shared storage medium are permitted, while writes are denied for expired leases. When a successful heartbeat is generated for a primary storage controller, the lease window is extended for the primary storage controller from the time of a previous heartbeat. Accordingly, a prolonged stall between successive heartbeats by the primary storage controller will result in the newly extended lease being expired at the time it is granted. This scheme prevents a split brain scenario from occurring when a secondary storage controller takes over as the new primary storage controller in response to detecting the stall.

What is claimed is: 
     
       1. A storage system comprising:
 a plurality of storage controllers configured to couple to one or more client computer systems via one or more data communications links; and 
 one or more storage devices coupled to the plurality of storage controllers; 
 wherein a first storage controller of the plurality of storage controllers is configured to:
 generate heartbeats on intervals of a first period of time; 
 issue, for a first portion of the storage devices, a lease responsive to generating a current heartbeat; 
 determine if a lease is valid; 
 responsive to determining that the lease is valid, perform a state-changing operation on the first portion of the storage devices; 
 detect a plurality of pending state-changing operations targeting the first portion of the storage devices; 
 determine a maximum number of state-changing operations that can be performed without causing data loss due to a failure of a fencing mechanism; and 
 perform a first number of state-changing operations on the first portion of the storage devices responsive to determining the particular lease is valid, wherein the first number is less than or equal to the maximum number; and 
 
 wherein a second storage controller of the plurality of storage controllers is configured to:
 generate heartbeats on intervals of a second period of time; 
 issue, for a second portion of the storage devices, a lease responsive to generating a current heartbeat; 
 determine if a lease is valid; and 
 responsive to determining that the lease is valid, perform a state-changing operation on the second portion of the storage devices; and 
 
 wherein the first storage controller and the second storage controller are configured to perform non-state-changing operations on both portions of the storage devices. 
 
     
     
       2. The storage system as recited in  claim 1 , wherein the first storage controller is further configured to prevent the state-changing operation from being performed on the first portion of the storage devices without a valid lease, and wherein the second storage controller does not have a valid lease on the first portion of the storage devices and the second storage controller cannot perform any state-changing operations on the first portion of the storage devices. 
     
     
       3. The storage system as recited in  claim 2 , wherein the first storage controller is further configured to:
 recheck if the particular lease is valid responsive to performing the first number of state-changing operations on the first portion of the storage devices; and 
 perform a second number of state-changing operations on the first portion of the storage devices responsive to determining the particular lease is valid, wherein the second number is less than or equal to the maximum number. 
 
     
     
       4. The storage system as recited in  claim 1 , wherein the first storage controller is a primary storage controller for the first portion of the storage devices, and wherein a second storage controller of the plurality of storage controllers is configured to take over as a new primary storage controller for the first portion of the storage devices responsive to detecting that the first storage controller has not generated a heartbeat for a particular period of time. 
     
     
       5. The storage system as recited in  claim 4 , wherein responsive to taking over as the new primary storage controller for the first portion of the storage devices, the second storage controller is configured to:
 generate heartbeats on intervals of the first period of time; 
 issue a new lease for the first portion of the storage devices responsive to generating a heartbeat, wherein a beginning of the new lease is calculated from a prior heartbeat and has a duration of a third period of time; and 
 determine if a particular lease is valid prior to performing a state-changing operation on the first portion of the storage devices. 
 
     
     
       6. The storage system as recited in  claim 1 , wherein the second storage controller is further configured to prevent state-changing operations from being performed on the second portion of the storage devices without a valid lease, and wherein the first storage controller does not have a valid lease on the second portion of the storage devices and the first storage controller cannot perform any state-changing operations on the second portion of the storage devices. 
     
     
       7. The storage system as recited in  claim 1 , further comprising:
 sending, from the first storage controller to the second storage controller, heartbeats via a data communications bus between the first storage controller and the second storage controller; and 
 sending, from the second storage controller to the first storage controller, heartbeats via a data communications bus between the first storage controller and the second storage controller. 
 
     
     
       8. A plurality of storage controllers:
 wherein each storage controller is configured to couple to one or more client computer systems via one or more data communications links and each storage controller is coupled to a plurality of storage devices; 
 wherein a first storage controller is configured to:
 generate heartbeats on intervals of a first period of time; 
 issue, for a first portion of the storage devices, a lease responsive to generating a current heartbeat; 
 determine if a lease is valid; 
 responsive to determining that the lease is valid, perform a state-changing operation on the first portion of the storage devices; 
 detect a plurality of pending state-changing operations targeting the first portion of the storage devices; and 
 determine a maximum number of state-changing operations that can be performed without causing data loss due to a failure of a fencing mechanism; and 
 perform a first number of state-changing operations on the first portion of the storage devices responsive to determining the particular lease is valid, wherein the first number is less than or equal to the maximum number; and 
 
 wherein a second storage controller is configured to:
 generate heartbeats on intervals of a second period of time; 
 issue, for a second portion of the storage devices, a lease responsive to generating a current heartbeat; 
 determine if a lease is valid; and 
 responsive to determining that the lease is valid, perform a state-changing operation on the second portion of the storage devices; and 
 
 wherein the first storage controller and the second storage controller are configured to perform non-state-changing operations on both portions of the storage devices. 
 
     
     
       9. The plurality of storage controllers as recited in  claim 8 , wherein the first storage controller is further configured to prevent the state-changing operation from being performed on the first portion of the storage devices without a valid lease, and wherein the second storage controller does not have a valid lease on the first portion of the storage devices and the second storage controller cannot perform any state-changing operations on the first portion of the storage devices. 
     
     
       10. The plurality of storage controllers as recited in  claim 9 , wherein the first storage controller is further configured to:
 recheck if the particular lease is valid responsive to performing the first number of state-changing operations on the first portion of the storage devices; and 
 perform a second number of state-changing operations on the first portion of the storage devices responsive to determining the particular lease is valid, wherein the second number is less than or equal to the maximum number. 
 
     
     
       11. The plurality of storage controllers as recited in  claim 8 , wherein the second storage controller is further configured to prevent state-changing operations from being performed on the second portion of the storage devices without a valid lease, and wherein the first storage controller does not have a valid lease on the second portion of the storage devices and the first storage controller cannot perform any state-changing operations on the second portion of the storage devices. 
     
     
       12. The plurality of storage controllers as recited in  claim 8 , wherein:
 the first storage controller is configured to send, to the second storage controller, heartbeats via a data communications bus between the first storage controller and the second storage controller; and 
 the second storage controller is configured to send, to the first storage controller, heartbeats via a data communications bus between the first storage controller and the second storage controller. 
 
     
     
       13. A method comprising:
 generating, by a first storage controller, heartbeats on intervals of a first period of time; 
 issuing, by the first storage controller for a first portion of the storage devices, a lease responsive to generating a current heartbeat; 
 determining, by the first storage controller, if a lease is valid; 
 responsive to determining that the lease is valid, perform by the first storage controller, a state-changing operation on the first portion of the storage devices; 
 generating, by a second storage controller, heartbeats on intervals of a second period of time; 
 issuing, by the second storage controller for a second portion of the storage devices, a lease responsive to generating a current heartbeat; 
 determining, by the second storage controller, if a lease is valid; 
 responsive to determining that the lease is valid, perform by the second storage controller, a state-changing operation on the second portion of the storage devices; 
 detecting, by the first storage controller, a plurality of pending state-changing operations targeting the first portion of the storage devices; 
 determining, by the first storage controller, a maximum number of state-changing operations that can be performed without causing data loss due to a failure of a fencing mechanism; and 
 performing, by the first storage controller, a first number of state-changing operations on the first portion of the storage devices responsive to determining the particular lease is valid, wherein the first number is less than or equal to the maximum number; 
 wherein the first storage controller and the second storage controller are configured to perform non-state-changing operations on both portions of the storage devices. 
 
     
     
       14. The method as recited in  claim 13 , further comprising preventing, by the first storage controller, the state-changing operation from being performed on the first portion of the storage devices without a valid lease, and wherein the second storage controller does not have a valid lease on the first portion of the storage devices and the second storage controller cannot perform any state-changing operations on the first portion of the storage devices. 
     
     
       15. The method as recited in  claim 14 , further comprising:
 rechecking, by the first storage controller, if the particular lease is valid responsive to performing the first number of state-changing operations on the first portion of the storage devices; and 
 performing, by the first storage controller, a second number of state-changing operations on the first portion of the storage devices responsive to determining the particular lease is valid, wherein the second number is less than or equal to the maximum number. 
 
     
     
       16. The method as recited in  claim 13 , wherein the second storage controller is further configured to prevent state-changing operations from being performed on the second portion of the storage devices without a valid lease, and wherein the first storage controller does not have a valid lease on the second portion of the storage devices and the first storage controller cannot perform any state-changing operations on the second portion of the storage devices. 
     
     
       17. The method as recited in  claim 13 , further comprising:
 sending, from the first storage controller to the second storage controller, heartbeats via a data communications bus between the first storage controller and the second storage controller; and 
 sending, from the second storage controller to the first storage controller, heartbeats via a data communications bus between the first storage controller and the second storage controller.