Load distribution across superclusters

Abstract

Examples described herein relate to apparatuses and methods for managing communications within a supercluster or across superclusters, including a first supercluster having a plurality of first machines and a publish-subscribe (Pub-Sub) channel to which each of the plurality of first machines is subscribed. A second supercluster has a plurality of second machines and a bridge between the first supercluster and the second supercluster. A first machine is configured to receive, via the bridge, an availability status and resource allocation information of each second machine and publish, on the Pub-Sub channel of the first supercluster, the availability status and the resource allocation information.

What is claimed is: 
     
       1. A supercluster, comprising:
 a plurality of machines; 
 a publish-subscribe (Pub-Sub) channel to which each of the plurality of machines is subscribed, 
 wherein each of the plurality of machines is configured to:
 publish availability status and resource allocation information to the Pub-Sub channel; 
 receive availability status and resource allocation information of at least another one of the plurality of machines via the Pub-Sub channel; 
 forward a request received from a client for content data to the another one of the plurality of machines based on the availability status and the resource allocation information on the Pub-Sub channel. 
 
 
     
     
       2. The supercluster of  claim 1 , wherein
 the availability status indicates whether each of the plurality of machines is available to store content data; and 
 the resource allocation information identifies the content data stored by each of the plurality of machines. 
 
     
     
       3. The supercluster of  claim 1 , wherein the Pub-Sub channel comprises a multicast address. 
     
     
       4. The supercluster of  claim 1 , wherein forwarding the request received from the client for the content data to the another one of the plurality of machines based on the availability status and the resource allocation information on the Pub-Sub channel comprises:
 receive, by a first machine of the plurality of machines, the request for the content data from the client; 
 determine, by the first machine, that the first machine does not store the content data; 
 determine, based on the availability status and the resource allocation information on the Pub-Sub channel, that the content data is stored on a second machine of the plurality of machines; and 
 forward the request to the second machine. 
 
     
     
       5. The supercluster of  claim 1 , wherein a first machine of the plurality of machines owns content data owned by a second machine of the plurality of machine based on the availability status and the resource allocation information on the Pub-Sub channel. 
     
     
       6. The supercluster of  claim 5 , wherein, to own the content data currently owned by the second machine, the first machine:
 identifies, based on the availability status and the resource allocation information on the Pub-Sub channel, the content data owned by the second machine; 
 publishes, to the Pub-Sub channel, availability to own the content data owned by the second machine; 
 receives, from the second machine via the Pub-Sub channel, authorization from the second machine; 
 owning the content data owned by the second machine; 
 receives, from a third machine of the plurality of machines, a relayed request for the content data; and 
 provides the content data in response to the relayed request. 
 
     
     
       7. The supercluster of  claim 5 , wherein, to own the content data currently owned by the second machine, the first machine:
 determines, based on the availability status and the resource allocation information on the Pub-Sub channel, that a number of second machines currently storing the content data is below a threshold; 
 publishes, to the Pub-Sub channel, availability to own the content data; 
 receives, via the Pub-Sub channel, authorization from the second machines currently storing the content data; 
 owns the content data; 
 receives, from a third machine of the plurality of machines, a relayed request for the content data; and 
 provides the content data in response to the relayed request. 
 
     
     
       8. The supercluster of  claim 1 , wherein a first machine of the plurality of machines is configured to establish a bridge to a second machine in another supercluster, wherein:
 the another supercluster neighbors the supercluster; and 
 the bridge is configured to relay, to the first machine, an availability status and resource allocation information of each of at least some machines of the another supercluster. 
 
     
     
       9. The supercluster of  claim 8 , wherein:
 the first machine is configured to determine, based on the availability status of the plurality of machines on the Pub-Sub channel, that a number of unavailable machines of the plurality of machines exceeds a threshold; and 
 the bridge is established in response to determining that the number of unavailable machines exceeds the threshold. 
 
     
     
       10. The supercluster of  claim 8 , wherein the bridge comprises a Transmission Control Protocol (TCP) connection. 
     
     
       11. The supercluster of  claim 8 , wherein the first machine is configured to publish the availability status and the resource allocation information of the at least some machines of the other supercluster on the Pub-Sub channel. 
     
     
       12. The supercluster of  claim 11 , wherein a third machine of the supercluster is configured to:
 receive the request from the client for the content data; 
 determine, based on the resource allocation information of the at least some machines of the another supercluster on the Pub-Sub channel, that the content data is stored by a fourth machine of the another supercluster; and 
 forward the request to the fourth machine. 
 
     
     
       13. A method for distributing loads across a plurality of machines of a supercluster, comprising:
 publishing, by each of the plurality of machines of the supercluster, availability status and resource allocation information to a publish-subscribe (Pub-Sub) channel, wherein each of the plurality of machines is subscribed to the Pub-Sub channel; 
 receiving availability status and resource allocation information of at least another one of the plurality of machines via the Pub-Sub channel; and 
 forwarding a request received from a client for content data to the another one of the plurality of machines based on the availability status and the resource allocation information on the Pub-Sub channel. 
 
     
     
       14. The method of  claim 13 , wherein
 the availability status indicates whether each of the plurality of machines is available to store content data; and 
 the resource allocation information identifies the content data stored by each of the plurality of machines. 
 
     
     
       15. The method of  claim 13 , wherein forwarding the request received from the client for the content data to the another one of the plurality of machines based on the availability status and the resource allocation information on the Pub-Sub channel comprises:
 receiving, by a first machine of the plurality of machines, the request for the content data from the client; 
 determining, by the first machine, that the first machine does not store the content data; 
 determining, based on the availability status and the resource allocation information on the Pub-Sub channel, that the content data is stored on a second machine of the plurality of machines; and 
 forwarding the request to the second machine. 
 
     
     
       16. The method of  claim 13 , wherein a first machine of the plurality of machines is configured to establish a bridge to a second machine in another supercluster, wherein:
 the another supercluster neighbors the supercluster; and 
 the bridge is configured to relay, to the first machine, an availability status and resource allocation information of each of at least some machines of the another supercluster. 
 
     
     
       17. The method of  claim 16 , wherein:
 the first machine is configured to determine, based on the availability status of the plurality of machines on the Pub-Sub channel, that a number of unavailable machines of the plurality of machines exceeds a threshold; and 
 the bridge is established in response to determining that the number of unavailable machines exceeds the threshold. 
 
     
     
       18. The method of  claim 16 , wherein the bridge comprises a Transmission Control Protocol (TCP) connection. 
     
     
       19. The method of  claim 16 , wherein the first machine is configured to publish the availability status and the resource allocation information of the at least some machines of the other supercluster on the Pub-Sub channel. 
     
     
       20. A non-transitory computer-readable medium having computer-readable instructions such that, when executed, cause a processor of each of a plurality of machines of a supercluster to:
 publish availability status and resource allocation information to a publish-subscribe (Pub-Sub) channel, wherein each of the plurality of machines is subscribed to the Pub-Sub channel; 
 receive availability status and resource allocation information of at least another one of the plurality of machines via the Pub-Sub channel; and 
 forward a request received from a client for content data to the another one of the plurality of machines based on the availability status and the resource allocation information on the Pub-Sub channel.