High reliability fault tolerant computer architecture

Abstract

A fault tolerant computer system and method are disclosed. The system may include a plurality of CPU nodes, each including: a processor and a memory; at least two IO domains, wherein at least one of the IO domains is designated an active IO domain performing communication functions for the active CPU nodes; and a switching fabric connecting each CPU node to each IO domain. One CPU node is designated a standby CPU node and the remainder are designated as active CPU nodes. If a failure, a beginning of a failure, or a predicted failure occurs in an active node, the state and memory of the active CPU node are transferred to the standby CPU node which becomes the new active CPU node. If a failure occurs in an active IO domain, the communication functions performed by the failing active IO domain are transferred to the other IO domain.

1 .- 23 . (canceled) 
     
     
         24 . A method of performing input/output (IO) domain failover in a fault tolerant computer system having a plurality of CPU node sand a plurality of IP domains; the method comprising:
 designating at least one of the active IO domains as a primary IO domain;   performing active communication functions using the primary IO domain for the active CPU nodes;   connecting each CPU node to each IO domain using a switching fabric, wherein each IO domain is connected to the switching fabric by one or more links;   enabling a failure trigger for each switching fabric control component in each IO domain to detect a failure of the primary or secondary IO domain or a component thereof, wherein the failure trigger comprises one or more of link-down errors, uncorrectable and fatal errors, and software triggers;   upon failure trigger occurring, stopping drivers using the failing IO domain or failing IO component; and   continuing to use the remaining operational IO domains and IO components.   
     
     
         25 . The method of  claim 24  further comprising isolating the failing IO domain or IO component upon failure trigger occurring. 
     
     
         26 . The method of  claim 24  further comprising running a provisioning service for each active IO domain, wherein each provisioning service communicates with the provisioning service of the other active IO domain to form a unified hierarchy of physical and/or virtual functions. 
     
     
         27 . The method of  claim 24  further comprising generating an interrupt for each CPU node, wherein the interrupt communicates that the IO domain or component there has failed to each CPU node. 
     
     
         28 . The method of  claim 24 , wherein each IO domain further comprises a set of IO devices, wherein the set of IO devices comprises the component of the primary IO domain, wherein each IO device comprises one or more physical functions and/or virtual functions, wherein one or more physical and/or virtual functions in one IO domain are shareable. 
     
     
         29 . The method of  claim 24 , wherein each CPU node and IO domain are configured to be replaced without affecting applications executing on one or more of the other CPU nodes and IO domain. 
     
     
         30 . The method of  claim 24  further comprising executing an operating system and one or more customer applications using the one or more of the active CPU nodes. 
     
     
         31 . The method of  claim 30  further comprising allocating the set of IO devices and the one or more physical and/or virtual functions to one or more CPU nodes and one or more of two switching fabric control components 
     
     
         32 . The method of  claim 26  further comprising defining one or more sub-hierarchies assignable to ports of the one or more CPU nodes. 
     
     
         33 . The method of  claim 24 , wherein each IO domain further comprises a set of IO devices, the method further comprising partitioning one or more of the set of IO devices and the virtual functions among a set of physical CPU nodes, the set of physical CPU nodes comprising the active CPU node and the standby CPU node 
     
     
         34 . The method of  claim 24  further comprising running one or more management engine instances using a management processor in each IO domain, wherein each management engine queries the switching fabric control components connected to the respective management engine to obtain an enumerated hierarchy of physical and/or virtual functions on a per control component basis, wherein each management engine merges enumerated per-component hierarchies into a per-domain hierarchy of physical and/or virtual functions within the IO domain associated with each management engine. 
     
     
         35 . The method of  claim 24  further comprising controlling communication through a management processor of an active IO domain through the switching fabric. 
     
     
         36 . The method of  claim 24  further comprising attempting to bring the failing IO domain or the failing IO component back into service by asserting an independent reset to the failing IO domain or failing IO component. 
     
     
         37 . A fault tolerant computer system comprising:
 a plurality of CPU nodes, each CPU node comprising a processor and a memory, wherein one of the CPU nodes is designated a standby CPU node and the remainder are designated as active CPU nodes;   at least two IO domains, wherein either IO domain can perform all of the communication functions for the active CPU nodes, wherein each IO domain comprises a set of IO devices; and   a switching fabric connecting each CPU node to each IO domain,   wherein if one of a failure, a beginning of a failure, and a predicted failure occurs in an active IO domain, termed the failing active IO domain, the communication functions performed by the failing active IO domain are continued by another IO domain.   
     
     
         38 . The system of  claim 37  wherein if one of a failure, a beginning of a failure, and a predicted failure occurs in an active IO domain, remove or isolate the active IO domain or component thereof using one or more switching fabric control components. 
     
     
         39 . The system of  claim 37  wherein if one of a failure, a beginning of a failure, and a predicted failure occurs in an active IO domain, the active IO domain is deprovisioned and a secondary IP domain is used in lieu thereof. 
     
     
         40 . The system of  claim 37  wherein if one of a failure, a beginning of a failure, and a predicted failure occurs in an active IO domain, isolate the failing IO domain upon failure trigger occurring. 
     
     
         41 . The system of  claim 37  further comprising a provisioning service for each active IO domain, wherein each provisioning service communicates with the provisioning service of the other active IO domain to form a unified hierarchy of physical and/or virtual functions. 
     
     
         42 . The system of  claim 38 , wherein each IO domain further comprises a management processor. 
     
     
         43 . The system of  claim 42 , wherein the management processor of an active IO domain controls communication through the switching fabric. 
     
     
         44 . The system of  claim 37  wherein upon the occurrence of one of the failure, the beginning of the failure, and the predicted failure, an interrupt for each CPU node, wherein the interrupt communicates that the IO domain or component there has failed to each CPU node. 
     
     
         45 . The system of  claim 37  wherein each IO domain further comprises a set of IO devices, wherein each IO device comprises one or more physical functions and/or virtual functions, wherein one or more physical and/or virtual functions in one IO domain are shareable. 
     
     
         46 . The fault tolerant computer system of  claim 37 , wherein each CPU node further comprises a communication interface in communication with the switching fabric. 
     
     
         47 . The fault tolerant computer system of  claim 37 , wherein each IO domain comprises at least two switching fabric control components, each switching fabric control component in communication with the switching fabric.