Distributed device status in a clustered system environment

Abstract

Problem ownership and status propagation in a clustered system environment must be coordinated in a way that avoids redundant and/or conflicting recovery efforts. IOP managers which overcome these problems are disclosed. A primary IOP manager is one which has problem ownership for a particular IOP. There is only one primary IOP manager for each IOP. A secondary IOP manager is one which resides upon a computer system that shares the resources of a particular IOP or IOPs, but nevertheless does not have problem ownership for that particular IOP(s). There are one or more secondary IOP managers for a given IOP. When a primary IOP manager is informed of a problem in a device attached to a shared IOP, it updates resident system management information to indicate the new status of the subject device, informs its local system management of the problem so that corrective actions may be taken, and sends the status information to the secondary IOP managers via the IOP itself. When a secondary IOP manager is informed of a problem with a device attached to a shared IOP, it updates system management information to indicate the new status of the subject device and sends the status information to the primary IOP manager and other secondary IOP managers via the IOP itself. The secondary IOP manager does not, however, inform its local system management that corrective actions are required.

What is claimed is: 
     
       1. An apparatus for communicating error and status information, said apparatus comprising: first, second, and third computer systems, said first, second, and third computer systems being interconnected via a shared bus;   first, second, and third system managers, said first system manager residing on said first computer system, said second system manager residing on said second computer system, and said third system manager residing on said third computer system;   a primary IOP manager and first and second secondary IOP managers, said primary IOP manager residing on said first computer system, said first secondary IOP manager residing on said second computer system, and said second secondary IOP manager residing on said third computer system;   a plurality of IOPs, each of said plurality of IOPs being capable of communicating said error and status information to said first, second, and third computer systems;   means for communicating said error and status information from a first IOP to said primary IOP manager, said first IOP being one of said plurality IOPs;   means for communicating said error and status information from said primary IOP manager to said first system manager;   means for communicating said error and status information from said primary IOP manager to said first secondary IOP manager and said second secondary IOP manager, said error and status information being communicated via said first IOP; and   means for communicating said error and status information from said first secondary IOP manager to said second system manager and means for communicating said error and status information from said second secondary IOP manager to said third system manager.   
     
     
       2. The apparatus of claim 1 wherein said means for communicating said error and status information from said primary IOP manager to said first system manager further comprises: means for logging said error information into a first system error log, said first system error log residing on said first computer system.   
     
     
       3. The apparatus of claim 2 wherein said means for communicating said error and status information from said first secondary IOP manager to said second system manager and means for communicating said error and status information from said second secondary IOP manager to said third system manager further comprises: means for logging said error information into a second and third system error logs, said second system error log residing on said second computer system and said third system error log residing on said third computer system.   
     
     
       4. An apparatus for communicating error and status information, said apparatus comprising: first, second, and third computer systems;   first, second, and third system managers, said first system manager residing on said first computer system, said second system manager residing on said second computer system, and said third system manager residing on said third computer system;   a primary IOP manager and first and second secondary IOP managers, said primary IOP manager residing on said first computer system, said first secondary IOP manager residing on said second computer system, and said second secondary IOP manager residing on said third computer system;   a plurality of IOPs, each of said plurality of IOPs being capable of communicating status and error information to said first, second, and third computer systems;   means for communicating said error and status information from a first IOP to said first secondary IOP manager, said first IOP being one of said plurality of IOPs;   means for communicating said error and status information from said first secondary IOP manager to said second system manager;   means for communicating said error and status information from said first secondary IOP manager to said primary IOP manager and said second secondary IOP manager, said error and status information being communicated via said first IOP; and   means for communicating said error and status information from said primary IOP manager to said first system manager and means for communicating said error and status information from said second secondary IOP manager to said third system manager.   
     
     
       5. The apparatus of claim 4 wherein said means for communicating said error and status information from said first secondary IOP manager to said second system manager further comprises: means for logging said error information into a second system error log, said second system error log residing on said second computer system.   
     
     
       6. The apparatus of claim 5 wherein said means for communicating said error and status information from said primary IOP manager to said first system manager and means for communicating said error and status information from said second secondary IOP manager to said third system manager further comprises: means for logging said error information into first and third system logs, said first system error log residing on said first computer system and said third system error log residing on said third computer system.   
     
     
       7. The apparatus of claim 4 further comprises: means for sending acknowledgment information from said primary IOP manager to said first secondary IOP manager.   
     
     
       8. The apparatus of claim 7 wherein said means for communicating said error and status information from said first secondary IOP manager to said primary IOP manager and said second secondary IOP manager further comprises: means for setting a timer; and   means for waiting for said acknowledgment information and for expiration of said timer.   
     
     
       9. A method for communicating error and status information, said method comprising the steps of: communicating said error and status information from a first IOP to a primary IOP manager, said first IOP being one of a plurality of IOPs, said primary IOP manager residing on a first computer system;   communicating said error and status information from said primary IOP manager to a first system manager, said first system manager residing on said first computer system;   communicating said error and status information from said primary IOP manager to a first secondary IOP manager and a second secondary IOP manager, said first secondary IOP manager residing on a second computer system and said second secondary IOP manager residing on a third computer system, said error and status information being communicated via said first IOP; and   communicating said error and status information from said first secondary IOP manager to a second system manager and communicating said error and status information from said second secondary IOP manager to a third system manager, said second system manager residing on said second computer system and said third system manager residing on said third computer system.   
     
     
       10. The method of claim 9 wherein said step of communicating said error and status information from said primary IOP manager to said first system manager further comprises the step of: logging said error information into a first system error log, said first system error log residing on said first computer system.   
     
     
       11. The method of claim 10 wherein said step of communicating said error and status information from said first secondary IOP manager to said second system manager and communicating said error and status information from said second secondary IOP manager to said third system manager further comprises the step of: logging said error information into a second and third system error logs, said second system error log residing on said second computer system and said third system error log residing on said third computer system.   
     
     
       12. A method for communicating error and status information amongst first, second, and third computer systems, said method comprising the steps of: communicating said error and status information from an IOP to a first secondary IOP manager, said IOP being one of a plurality of IOPs, said first secondary IOP manager residing on said second computer system;   communicating said error and status information from said first secondary IOP manager to a second system manager, said second system manager residing on said second computer system;   communicating said error and status information from said first secondary IOP manager to a primary IOP manager and a second secondary IOP manager, said primary IOP manager residing on said first computer system and said second secondary IOP manager residing on said third computer system, said error and status information being communicated via said IOP; and   communicating said error and status information from said primary IOP manager to a first system manager and communicating said error and status information from said second secondary IOP manager to a third system manager, said first system manager residing on said first computer system and said third system manager residing on said third computer system.   
     
     
       13. The method of claim 12 wherein said step of communicating said error and status information from said first secondary IOP manager to said second system manager further comprises the step of: logging said error information into a second system error log, said second system error log residing on said second computer system.   
     
     
       14. The method of claim 13 wherein said step of communicating said error and status information from said primary IOP manager to said first system manager and communicating said error and status information from said second secondary IOP manager to said third system manager further comprises the step of: logging said error and status information into first and third system logs, said first system error log residing on said first computer system and said third system error log residing on said third computer system.   
     
     
       15. The method of claim 12 further comprises the step of: sending acknowledgment information from said primary IOP manager to said first secondary IOP manager.   
     
     
       16. The method of claim 15 wherein said step of communicating said error and status information from said first secondary IOP manager to said primary IOP manager and said second secondary IOP manager further comprising the step of: setting a timer;   waiting for said acknowledgment information and for expiration of said timer; and   recommunicating said error and status information to said second system manager upon said expiration of said timer.   
     
     
       17. An apparatus for communicating error and status information amongst the nodes of a network of nodes, said apparatus comprising: first, second, and third nodes, said first, second, and third nodes being interconnected via a network;   first, second, and third system managers, said first system manager residing on said first node, said second system manager residing on said second node, and said third system manager residing on said third node;   a primary communications manager and first and second secondary communications managers, said primary communications manager residing on said first node, said first secondary communications manager residing on said second node, and said second secondary communications manager residing on said third node;   a plurality of shared devices, said shared devices being capable of communicating error and status information to said first, second, and third nodes;   means for communicating said error and status information from a first shared device to said primary communications manager, said first shared device being one of said plurality of shared devices;   means for communicating said error and status information from said primary communications manager to said first system manager;   means for communicating said error and status information from said primary communications manager to said first secondary communications manager and said second secondary communications manager, said error and status information being communicated via said first shared device; and   means for communicating said error and status information from said first secondary communications manager to said second system manager and means for communicating said error and status information from said second secondary communications manager to said third system manager.   
     
     
       18. An apparatus for communicating error and status information, said apparatus comprising: first, second, and third nodes;   first, second, and third system managers, said first system manager residing on said first node, said second system manager residing on said second node, and said third system manager residing on said third node;   a primary communications manager and first and second secondary communications managers, said primary communications manager residing on said first node, said first secondary communications manager residing on said second node, and said second secondary communications manager residing on said third node;   a plurality of shared devices, each of said plurality of shared devices being capable of communicating error and status information to said first, second, and third nodes;   means for communicating said error and status information from a first shared device to said first secondary communications manager, said first shared device being one of said plurality of shared devices;   means for communicating said error and status information from said first secondary communications manager to said second system manager;   means for communicating said error and status information from said first secondary communications manager to said primary communications manager and said second secondary communications manager, said error and status information being communicated via said first shared device; and   means for communicating said error and status information from said primary communications manager to said first system manager and means for communicating said error and status information from said second secondary communications manager to said third system manager.