Controller communications over an always-on controller interconnect

Abstract

A controller interconnect structure within a RAID disk array enables continuous low latency/high bandwidth communications between a plurality of controller pairs within the array. Mirror buses carry high speed mirror traffic between mirrored controllers performing mirrored memory operations. Loop buses carry inter-processor communications and other traffic between controller pairs coupled together in a controller loop. Benefits of the interconnect structure include an ability to support continued controller communications and online disk array operations under various failure and repair conditions that might otherwise render a disk array inoperable. In addition, the controller interconnect structure provides for easy expansion of the number of controllers within disk arrays as arrays continue to be scaled up in size to meet increasing storage demands from user host systems.

1 . A disk array comprising: 
 at least two controller pairs, each controller pair having a first controller coupled to a second controller through a mirror bus; and    a controller loop coupling all controllers through a plurality of loop buses.    
     
     
         2 . A disk array as recited in  claim 1 , wherein each controller in the controller loop is coupled to a first logically adjacent controller through a first loop bus and to a second logically adjacent controller through a second loop bus.  
     
     
         3 . A disk array as recited in  claim 1 , wherein the controller loop further comprises: 
 a first controller row having all first controllers coupled as a row of controllers; and    a second controller row having all second controllers coupled as a row of controllers; and    wherein a beginning controller in the first controller row is coupled to an ending controller in the second controller row and an ending controller in the first controller row is coupled to a beginning controller in the second controller row.    
     
     
         4 . A disk array as recited in  claim 1 , wherein each bus is divided into a first half and a second half, the disk array further comprising: 
 a first interconnect to which each controller is operatively coupled, the first interconnect carrying the first half of each bus; and    a second interconnect to which each controller is operatively coupled, the second interconnect carrying the second half of each bus.    
     
     
         5 . A disk array as recited in  claim 4 , wherein the first interconnect and the second interconnect- are selected from a group of interconnects comprising: 
 a back plane; and    a cable.    
     
     
         6 . A disk array as recited in  claim 1 , wherein each controller comprises routing logic configured to route data along the controller loop.  
     
     
         7 . A disk array as recited in  claim 6 , wherein the routing logic comprises a programmable routing register configured to contain routing information usable by the routing logic.  
     
     
         8 . A disk array as recited in  claim 1 , wherein each controller comprises a hardware circuit configured to detect a failure in the controller loop and to reroute data to avoid the failure.  
     
     
         9 . A disk array as recited in  claim 8 , wherein the hardware circuit is further configured to modify contents of a routing register so that data is initially routed to avoid the failure.  
     
     
         10 . A disk array comprising: 
 at least two controller pairs, each controller pair having a first controller coupled to a second controller through a mirror bus; and    a controller loop coupling all controllers through a plurality of loop buses and at least one mirror bus.    
     
     
         11 . A controller interconnect structure in a disk array comprising: 
 a plurality of controllers;    a plurality of communications buses operatively coupling the plurality of controllers to one another, each communications bus partitioned into a first half and a second half;    a first interconnect to which each of the plurality of controllers is coupled, the first interconnect conveying the first half of the plurality of communications buses; and    a second interconnect to which each of the plurality of controllers is coupled, the second interconnect conveying the second half of the plurality of communications buses.    
     
     
         12 . A controller interconnect structure as recited in  claim 11 , wherein the plurality of controllers comprises a plurality of controller pairs, each controller pair having a first controller and a second controller.  
     
     
         13 . A controller interconnect structure as recited in  claim 12 , wherein the plurality of communications buses comprises: 
 between each controller pair, a mirror bus operatively coupling the first controller to the second controller; and    a loop bus operatively coupling each controller with two logically adjacent controllers such that all controllers form part of a continuous controller loop.    
     
     
         14 . A controller interconnect structure as recited in  claim 11 , wherein each controller further comprises routing logic configured to route data along the controller loop to a destination controller pair based on an address header and a routing instruction.  
     
     
         15 . A controller interconnect structure as recited in  claim 14 , wherein the routing logic comprises a programmable routing register configured to contain the routing instruction, the routing logic further configured to access the programmable routing register and route data along the controller loop to a destination controller pair based on the routing instruction.  
     
     
         16 . A controller interconnect structure as recited in  claim 11 , wherein each controller comprises a hardware circuit configured to detect a failure in the interconnect structure and to reroute data over the interconnect structure to avoid the failure.  
     
     
         17 . A controller interconnect structure as recited in  claim 16 , wherein the hardware circuit is further configured to modify contents of a routing register so that data is initially routed over the interconnect structure to avoid the failure.  
     
     
         18 . A controller interconnect structure as recited in  claim 11 , wherein the first interconnect and the second interconnect are selected from a group of interconnects comprising: 
 a back plane; and    a cable.    
     
     
         19 . A controller interconnect structure comprising: 
 a first controller level having at least two controller pairs, each controller pair having a first controller coupled to a second controller through a mirror bus; and    a first controller loop coupling all controllers on the first controller level through a plurality of loop buses.    
     
     
         20 . A controller interconnect structure as recited in  claim 19 , further comprising: 
 a second controller level having at least two controller pairs, each controller pair having a first controller coupled to a second controller through a mirror bus; and    a second controller loop coupling all controllers on the second controller level through a plurality of loop buses;    wherein, each first controller of each controller pair on the first controller level is coupled via a loop bus to a corresponding first controller of a corresponding controller pair on the second controller level, and each second controller of each controller pair on the first controller level is coupled via a loop bus to a corresponding second controller of a corresponding controller pair on the second controller level.    
     
     
         21 . A processor-readable medium comprising processor-executable instructions configured for: 
 determining a destination controller pair for a data packet based on a host address for the data packet and an array mapping of the host address to an array address; and    initially sending the data packet over a controller loop toward the destination controller pair in a first direction determined by a data header and a routing instruction.    
     
     
         22 . A processor-readable medium as recited in  claim 21 , wherein the determining further comprises attaching the data header to the data packet.  
     
     
         23 . A processor-readable medium as recited in  claim 21 , comprising further processor-executable instructions configured for: 
 detecting a failure in the controller loop; and    rerouting the data packet in a second direction toward the destination controller pair.    
     
     
         24 . A processor-readable medium as recited in  claim 23 , comprising further processor-executable instructions configured for: 
 based on the failure, reprogramming a routing register with new routing information; and    based on the new routing information, initially sending additional data packets in a direction over the controller loop that avoids the failure.    
     
     
         25 . A processor-readable medium as recited in  claim 24 , comprising further processor-executable instructions configured for: 
 sharing failure information with all controllers; and    based on the failure information, reprogramming routing registers on each controller with new routing information.    
     
     
         26 . A processor-readable medium comprising processor-executable instructions configured for: 
 receiving at a first controller, a data packet that is destined for a controller pair;    attaching a header to the data packet;    accessing from a routing register, a routing instruction associated with the header; and    based on the routing instruction, sending the data packet over a controller loop in a first direction to a first mirrored controller of the two mirrored controllers in the controller pair.    
     
     
         27 . A processor-readable medium as recited in  claim 26 , comprising further processor-executable instructions configured for determining the controller pair based on a host address for the data packet and an array mapping of the host address to an array address.  
     
     
         28 . A processor-readable medium as recited in  claim 26 , comprising further processor-executable instructions configured for: 
 detecting a failure in the controller loop; and    rerouting the data packet in a second direction to a second mirrored controller of the two mirrored controllers in the controller pair.    
     
     
         29 . A processor-readable medium as recited in  claim 28 , comprising further processor-executable instructions configured for: 
 based on the failure, reprogramming the routing register with new routing information; and    based on the new routing information, initially sending additional data packets in a direction over the controller loop that avoids the failure.    
     
     
         30 . A processor-readable medium comprising processor-executable instructions configured for: 
 receiving at a first controller in a first controller level, data that is destined for a second level controller pair located in a second controller level;    sending the data over a first level controller loop to a first level controller pair that corresponds to the second level controller pair; and    further sending the data from the first level controller pair to the second level controller pair via a loop bus that couples the first controller level to the second controller level.    
     
     
         31 . A processor-readable medium as recited in  claim 30 , comprising further processor-executable instructions configured for: 
 detecting a failure in a controller loop; and    rerouting data traveling over the controller loop from a first direction to a second direction to avoid the failure.    
     
     
         32 . A processor-readable medium as recited in  claim 31 , comprising further processor-executable instructions configured for: 
 based on the failure, programming routing registers on all controllers in the controller loop with routing information; and    based on the routing information, initially sending additional data in a direction over the controller loop that avoids the failure.    
     
     
         33 . A processor-readable medium as recited in  claim 31 , comprising further processor-executable instructions configured for: 
 sharing failure information with all controllers; and    based on the failure information, reprogramming routing registers on each controller with new routing information.    
     
     
         34 . A processor-readable medium comprising processor-executable instructions configured for: 
 receiving data at a first controller in a first controller level;    sending the data from the first controller to a second level controller pair via a loop bus that couples the first controller level to a second controller level; and    further sending the data over a second level controller loop to a destination controller pair in the second controller level.    
     
     
         35 . A method of routing data between controller pairs in a multi-controller disk array comprising: 
 determining a destination controller pair for a data packet based on a host address for the data packet and an array mapping of the host address to an array address; and    initially sending the data packet over a controller loop toward the destination controller pair in a first direction determined by a data header and a routing instruction.    
     
     
         36 . A method as recited is  claim 35 , further comprising: 
 detecting a failure in the controller loop; and    rerouting the data packet in a second direction toward the destination controller pair.    
     
     
         37 . A method of routing data between controller pairs in a multi-controller disk array comprising: 
 receiving at a first controller, a data packet that is destined for a controller pair;    attaching a header to the data packet;    accessing from a routing register, a routing instruction associated with the header; and    based on the routing instruction, sending the data packet over a controller loop in a first direction to a first mirrored controller of the two mirrored controllers in the controller pair.    
     
     
         38 . A method as recited in  claim 37 , further comprising determining the controller pair based on a host address for the data packet and an array mapping of the host address to an array address.  
     
     
         39 . A method of routing data between controller pairs in a multi-controller disk array comprising: 
 receiving at a first controller in a first controller level, data that is destined for a second level controller pair located in a second controller level;    sending the data over a first level controller loop to a first level controller pair that corresponds to the second level controller pair; and    further sending the data from the first level controller pair to the second level controller pair via a loop bus that couples the first controller level to the second controller level.    
     
     
         40 . A method of routing data between controller pairs in a multi-controller disk array comprising: 
 receiving data at a first controller in a first controller level;    sending the data from the first controller to a second level controller pair via a loop bus that couples the first controller level to a second controller level; and    further sending the data over a second level controller loop to a destination controller pair in the second controller level.