Processing high volume network data

Abstract

Disclosed are a system comprising a computer-readable storage medium storing at least one program, and a computer-implemented method for event messaging over a network. An identification module receives first data identifying consumer devices available to receive event messages linked to a topic. The identification module receives second data identifying a producer device available to provide event messages linked to the topic. A provisioning module links a plurality of values to respective consumer devices that are linked to the topic. A scheduler module accesses a first data message linked to the topic. The first data message includes a key value. The scheduler module provides the first data message to a selected one of the consumer devices based on a comparison of the key value and the plurality of values of the respective consumer devices.

What is claimed is: 
     
       1. A system comprising:
 at least one processor and executable instructions accessible on a computer-readable medium that, when executed, cause the at least one processor to perform operations comprising:
 receiving first data identifying consumer devices available to receive event messages linked to a topic included in a plurality of topics; 
 receiving second data identifying a producer device available to provide event messages linked to the topic; 
 linking a plurality of consumer values to the consumer devices, the plurality of consumer values being computed to form a logical ring of the computing devices, such that each computing device is represented by a consumer node on the logical ring, the logical ring corresponding to the topic based on the plurality of consumer values being generated specific to the topic; 
 accessing from the producer device a first data message linked to the topic, the first data message including a key value; 
 generating a value from the key value included in the first data message; 
 selecting a consumer device from the consumer devices by walking around the logical ring of consumer nodes starting at a point corresponding to the value generated from the key value until reaching a consumer value linked to the consumer device; and 
 providing the first data message to the selected consumer device, the consumer devices including a plurality of server machines over which a workload, to process the event messages, is distributed. 
 
 
     
     
       2. The system of  claim 1 , wherein the operations further comprise:
 generating the plurality of consumer values using a consistent hashing process. 
 
     
     
       3. The system of  claim 1 , wherein the first and second data are communicated by a hardware-implemented transport component, and the first data message is provided to the selected consumer device by a hardware-implemented transport using a transmission control protocol (TCP) connection. 
     
     
       4. The system of  claim 1 , wherein the first data includes one or more request messages, each request message comprising a consumer device identifier, an Internet Protocol (IP) address and port binding of a corresponding consumer device and a topic identifier of the topic. 
     
     
       5. The system of  claim 4 , wherein the operations further comprise generating the plurality of consumer values based on the corresponding consumer device identifier in response to receiving respective request messages, wherein the plurality of consumer values is generated by a hashing algorithm in a pseudo random manner that distributes the consumer nodes approximately evenly on the logical ring. 
     
     
       6. The system of  claim 1 , wherein consumer values of the plurality of consumer values linked to the respective consumer devices are independent of the producer device. 
     
     
       7. The system of  claim 1 , wherein the operations further comprise:
 monitoring for changes in availability of the consumer devices; and 
 based on a determination that a consumer device is unavailable, removing the unavailable consumer device from the logical ring such that data messages are provided to a next nearest consumer node based on a direction used for selecting consumer devices from the logical ring. 
 
     
     
       8. The system of  claim 1 , wherein the operations further comprise batching the first data message with one or more other event messages for transmission to the selected consumer device, wherein the first data message and the one or more other event messages are batched based on the first data message and the one or more other event messages being associated with the topic corresponding to the logical ring having the consumer device to which the first data message is provided. 
     
     
       9. The system of  claim 1 , wherein the operations further comprise compressing the first data message selectively based on the first data. 
     
     
       10. The system of  claim 1 , wherein the consumer devices linked to the plurality of consumer values reside in two or more datacenters. 
     
     
       11. The system of  claim 1 , wherein the second data identifies the producer device from a plurality of producer devices that each use a same hashing algorithm that is configured to generate the plurality of consumer values, and wherein the same hashing algorithm is specific to the topic. 
     
     
       12. A method comprising:
 receiving first data identifying consumer devices available to receive event messages linked to a topic included in a plurality of topics; 
 receiving second data identifying a producer device available to provide event messages linked to the topic; 
 linking a plurality of consumer values to the consumer devices, the plurality of consumer values being computed to form a logical ring of the computing devices, such that each computing device is represented by a consumer node on the logical ring, the logical ring corresponding to the topic based on the plurality of consumer values being generated specific to the topic; 
 accessing from the producer device a first data message linked to the topic, the first data message including a key value; 
 generating a value from the key value included in the first data message; 
 selecting a consumer device from the consumer devices by walking around the logical ring of consumer nodes starting at a point corresponding to the value generated from the key value until reaching a consumer value linked to the consumer device; and 
 providing the first data message to the selected consumer device, the consumer devices including a plurality of server machines over which a workload, to process the event messages, is distributed. 
 
     
     
       13. The method of  claim 12 , further comprising:
 generating the plurality of consumer values using a consistent hashing process. 
 
     
     
       14. The method of  claim 12 , wherein the first and second data are communicated by a hardware-implemented transport component, and the providing of the first data message is performed by a hardware-implemented transport using a transmission control protocol (TCP) connection. 
     
     
       15. The method of  claim 12 , wherein the first data includes one or more request messages, each request message comprising a consumer device identifier, an Internet Protocol (IP) address and port binding, and a topic identifier. 
     
     
       16. The method of  claim 15 , further comprising generating, in response to receiving respective request messages, the plurality of consumer values based on the corresponding consumer device identifier, wherein the plurality of consumer values is generated by a hashing algorithm in a pseudo random manner that distributes the consumer nodes approximately evenly on the logical ring. 
     
     
       17. The method of  claim 12 , wherein the second data identifies the producer device from a plurality of producer devices that each use a same hashing algorithm that is configured to generate the plurality of consumer values, and wherein the same hashing algorithm is specific to the topic. 
     
     
       18. A device-readable storage medium embodying instructions that, when executed by a device, cause the device to perform operations comprising:
 receiving first data identifying consumer devices available to receive event messages linked to a topic included in a plurality of topics; 
 receiving second data identifying a producer device available to provide event messages linked to the topic; 
 linking a plurality of consumer values to the consumer devices, the plurality of consumer values being computed to form a logical ring of the computing devices, such that each computing device is represented by a consumer node on the logical ring, the logical ring corresponding to the topic based on the plurality of consumer values being generated specific to the topic; 
 accessing from the producer device a first data message linked to the topic, the first data message including a key value; 
 generating a value from the key value included in the first data message; 
 selecting a consumer device from the consumer devices by walking around the logical ring of consumer nodes starting at a point corresponding to the value generated from the key value until reaching a consumer value linked to the consumer device; and 
 providing the first data message to the selected consumer device, the consumer devices including a plurality of server machines over which a workload, to process the event messages, is distributed. 
 
     
     
       19. The device-readable storage medium of  claim 18 , wherein the plurality of consumer values is generated by a hashing algorithm in a pseudo random manner that distributes the consumer nodes approximately evenly on the logical ring. 
     
     
       20. The device-readable storage medium of  claim 18 , wherein the operations further comprise:
 monitoring for changes in availability of the consumer devices; and 
 based on a determination that a consumer device is unavailable, removing the unavailable consumer device from the logical ring such that data messages are provided to a next nearest consumer node based on a direction used for selecting consumer devices from the logical ring.