Distributed control system for large scale geographically distributed data centers and methods for use therewith

Abstract

A system includes a plurality of agents to control managed equipment of a geographically distributed data center. A controller communicates management data with the plurality of agents via a management network. The controller and the plurality of agents are configured in a hierarchical topology. A first subset of the plurality of agents relay management data on behalf of a second subset of the plurality of agents. The controller is further configured to: communicate with a new agent to be provisioned into the plurality of agents; generate reconfiguration instructions that indicate an allocated parent agent of the plurality of agents to be associated with the new agent; and facilitate a provisioning of the new agent in accordance with the reconfiguration instructions, to connect the new agent to the allocated parent agent of the plurality of agents via the management network and disconnect the new agent from the controller.

What is claimed is: 
     
       1. A system for use with a geographically distributed data center having a plurality of managed equipment, the system comprising:
 a plurality of agents configured to control the managed equipment of the geographically distributed data center; and 
 a controller configured to communicate management data with the plurality of agents via a management network, wherein the controller and the plurality of agents are configured in a hierarchical topology, and wherein a first subset of the plurality of agents are configured to relay management data on behalf of a second subset of agents of the plurality of agents, and wherein the controller is further configured to: 
 communicate, via the management network, with a new agent to be provisioned into the plurality of agents; 
 generate reconfiguration instructions that indicate an allocated parent agent of the plurality of agents to be associated with the new agent; and 
 facilitate a provisioning the new agent in accordance with the reconfiguration instructions, wherein the facilitating the provisioning includes connecting the new agent to the allocated parent agent of the plurality of agents via the management network and disconnecting the new agent from the controller; 
 wherein, when a child agent of the second subset of agents is disconnected from its designed parent agent included in the first subset of agents, the child agent is configured to: 
 retry connection with its designated parent agent for a configurable amount of time; and 
 when child agent fails to connect with its designated parent agent during the configurable amount of time, the child agent attempts reconnection with the controller. 
 
     
     
       2. The system of  claim 1 , wherein the controller communicates directly via the management network with the first subset of the plurality of agents. 
     
     
       3. The system of  claim 2 , wherein the controller communicates indirectly via the management network with the second subset of the plurality of agents. 
     
     
       4. The system of  claim 3 , wherein the controller communicates indirectly via the management network with the second subset of the plurality of agents via ones of the first subset of the plurality of agents. 
     
     
       5. The system of  claim 1 , wherein at least one of the plurality of agents is configured to communicate directly with another agent of the plurality of agents to perform a coordinated distributed processing. 
     
     
       6. The system of  claim 5 , wherein the coordinated distributed processing includes a map-reduce function to produce aggregated data and wherein the at least one of the plurality of agents communicates the aggregated data to the controller. 
     
     
       7. The system of  claim 1 , wherein at least one of the plurality of agents is configured to perform a plurality of functions autonomously from the controller, wherein the plurality of functions include two or more of: initiating a connection to an upstream agent of the plurality of agents in the hierarchical topology, pulling state information from the upstream agent of the plurality of agents, attempting a reconnection with another agent of the plurality of agents, or auto-healing degraded portions of the management network under jurisdiction of the at least one of the plurality of agents. 
     
     
       8. The system of  claim 1 , wherein at least one of the plurality of agents is configured to perform a plurality of functions autonomously from the controller, wherein the plurality of functions include two or more of: initiating a connection to an upstream agent of the plurality of agents in the hierarchical topology, receiving tasks to be performed from the upstream agent of the plurality of agents, or receiving a notification that a state of the management network has changed. 
     
     
       9. The system of  claim 1 , wherein the controller determines the allocated parent agent for the new agent based on at least one of: a geographical proximity to the new agent the allocated parent agent or based on a latency triangulation between the new agent and the allocated parent agent. 
     
     
       10. A method for use in a geographically distributed data center having a plurality of managed equipment, the method comprising:
 providing a controller and a plurality of agents that communicate management data via a management network, wherein the plurality of agents control the managed equipment of the geographically distributed data center; wherein the controller and the plurality of agents are configured in a hierarchical topology, and wherein a first subset of the plurality of agents are configured to relay management data on behalf of a second subset of the plurality of agents; 
 providing a new agent to be provisioned into the plurality of agents; 
 initiating, via the new agent, a connection to the controller via the management network; 
 receiving, by the new agent, from the controller and via the management network, reconfiguration instructions that indicate an allocated parent agent of the plurality of agents; and 
 provisioning the new agent in accordance with the reconfiguration instructions, wherein the provisioning includes connecting the new agent to the allocated parent agent of the plurality of agents via the management network and disconnecting the new agent from the controller; 
 wherein, when a child agent of the second subset of agents is disconnected from its designed parent agent included in the first subset of agents, the child agent is configured to: 
 retry connection with its designated parent agent for a configurable amount of time; and 
 when child agent fails to connect with its designated parent agent during the configurable amount of time, the child agent attempts reconnection with the controller. 
 
     
     
       11. The method of  claim 10 , wherein the controller communicates directly via the management network with the first subset of the plurality of agents. 
     
     
       12. The method of  claim 11 , wherein the controller communicates indirectly via the management network with the second subset of the plurality of agents. 
     
     
       13. The method of  claim 12 , wherein the controller communicates indirectly via the management network with the second subset of the plurality of agents via ones of the first subset of the plurality of agents. 
     
     
       14. The method of  claim 10 , wherein at least one of the plurality of agents is configured to communicate directly with another agent of the plurality of agents to perform a coordinated distributed processing. 
     
     
       15. The method of  claim 14 , wherein the coordinated distributed processing includes a map-reduce function to produce aggregated data and wherein the at least one of the plurality of agents communicates the aggregated data to the controller. 
     
     
       16. The method of  claim 10 , wherein at least one of the plurality of agents is configured to perform a plurality of functions autonomously from the controller, wherein the plurality of functions include two or more of: initiating a connection to an upstream agent of the plurality of agents in the hierarchical topology, pulling state information from the upstream agent of the plurality of agents, attempting a reconnection with another agent of the plurality of agents, or auto-healing degraded portions of the management network under jurisdiction of the at least one of the plurality of agents. 
     
     
       17. The method of  claim 10 , wherein at least one of the plurality of agents is configured to perform a plurality of functions autonomously from the controller, wherein the plurality of functions include two or more of: initiating a connection to an upstream agent of the plurality of agents in the hierarchical topology, receiving tasks to be performed from the upstream agent of the plurality of agents, or receiving a notification that a state of the management network has changed. 
     
     
       18. The method of  claim 10 , wherein the controller determines the allocated parent agent for the new agent based on at least one of: a geographical proximity to the new agent the allocated parent agent or based on a latency triangulation between the new agent and the allocated parent agent.