Traffic Statistic Generation For Datacenters

Abstract

Systems and methods are disclosed for generating traffic statistics for a datacenter. Distributed, parallel processing may be used to generate traffic statistics from data sets about traffic in a datacenter. To reduce data sets from which such statistics are derived to manageable sizes and relevant processing times for distributed, parallel processing, traffic agents may be provided at end hosts in the datacenter. The traffic agents may summarize data traffic over large numbers of packets in terms of the various sockets over which they are transmitted. Reports on the various sockets may be sent by the various flow agents that monitor them to an analytic controller. The analytic controller may aggregate, provide flow-path information for, further reduce, and/or provision the resultant data for distributed parallel processing.

1 . A system comprising:
 a network comprising a plurality of nodes and a plurality of network connections between nodes, each node being a device programmed to perform packet switching in order to transfer packets throughout the network;   a plurality of endpoints, each endpoint of the plurality of endpoints being a computer system in data communication with the network and programmed to:
 transfer data packets from the each endpoint to other endpoints of the plurality of endpoints through the network; 
 gather traffic data for each tuple of a plurality of tuples, each tuple being the each endpoint and a different endpoint of the plurality of endpoints and the traffic data being a summary of the data packets sent to the different endpoint of the each tuple; 
 transfer a representation of the traffic data for each tuple of the plurality of tuples to a controller; 
   a controller that is in data communication with the network and includes a computer system programmed to:
 receive the representations of the traffic data from each endpoint of the plurality of endpoints; 
 convert each tuple of the plurality of tuples for each endpoint of the plurality of endpoints in the representation of the traffic data to a path according to topology information for the network to obtain post-activity data, the path including references to one or more nodes of the plurality of nodes; and 
 characterize traffic through nodes of the network according to the post-activity data. 
   
     
     
         2 . The system of  claim 1 , wherein each endpoint of the plurality of endpoints is further programmed to:
 generate the representation of the traffic data for each tuple of the plurality of tuples by summarizing data for a plurality of data packets transferred between the each endpoint and the second endpoint of the each tuple.   
     
     
         3 . The system of  claim 1 , wherein each endpoint of the plurality of endpoints is further programmed to:
 generate the representation of the traffic data for each tuple of the plurality of tuples by summarizing data for a plurality of data packets transferred between the each endpoint and the second endpoint of the each tuple over a plurality of different socket connections.   
     
     
         4 . The system of  claim 1 , wherein each endpoint of the plurality of endpoints is further programmed to:
 generate the representation of the traffic data for each tuple of the plurality of tuples by summarizing data for a plurality of data packets transferred between the each endpoint and the second endpoint of each tuple of the plurality of tuples over two or more socket connections during a group time period.   
     
     
         5 . The system of  claim 1 , wherein each endpoint of the plurality of endpoints is further programmed to transfer the data packets from the each endpoint to the other endpoints of the plurality of endpoints through the network using a connection-based packet switching approach. 
     
     
         6 . The system of  claim 5 , wherein the plurality of nodes of the network are programmed to route packets between endpoints of the plurality of endpoints according to a plurality of enforced paths, the topology information including a representation of the plurality of enforced paths. 
     
     
         7 . The system of  claim 1 , wherein each endpoint of the plurality of endpoints is further programmed to, for each tuple of the plurality of tuples, generate the representation of the traffic data by:
 generating a first representation of a first type of data flow in the traffic data; and   generating a second representation of a second type of data flow in the traffic data, the second type of data flow being of longer duration than the first type of data flow;   wherein each endpoint of the plurality of endpoints is further programmed to transfer both the first representation and the second representation to the controller.   
     
     
         8 . The system of  claim 1 , wherein each endpoint of the plurality of endpoints is further programmed to:
 for a portion of the tuples of the plurality of tuples, include a path through the network for each tuple of the portion in the representation of the traffic data;   wherein the controller is further programmed to characterize traffic through individual nodes of the network according to both the path through the network included in the representation of the traffic data for the each endpoint and the topology information.   
     
     
         9 . The system of  claim 1 , wherein each node of the plurality of nodes operates as at least one of a network switch and a router. 
     
     
         10 . The system of  claim 1 , wherein each endpoint of the plurality of endpoints comprises a computer system including a memory device, a processing device coupled to the memory device, and a network interface controller (NIC) coupled to the processing device. 
     
     
         11 . A method comprising:
 providing a network comprising a plurality of nodes and a plurality of network connections between nodes, each node being a device programmed to perform packet switching in order to transfer packets throughout the network;
 providing a plurality of endpoints in data communication with the network; 
 transferring, by a first endpoint of the plurality of endpoints, data packets from the first endpoint to other endpoints of the plurality of endpoints through the network; 
 gathering, by the first endpoint, traffic data for each tuple of a plurality of tuples, each tuple being the first endpoint and a different endpoint of the plurality of endpoints and the traffic data being a summary of the data packets sent to the different endpoint of the each tuple; 
 transferring, by the first endpoint, a representation of the traffic data for each tuple of the plurality of tuples to a controller; 
 providing a controller that is in data communication with the network and including a computer system; 
 receiving, by the controller, the representations of the traffic data from each endpoint of the plurality of endpoints; 
 converting, by the controller, each tuple of the plurality of tuples for each endpoint of the plurality of endpoints in the representation of the traffic data to a path according to topology information for the network to obtain post-activity data, the path including references to one or more nodes of the plurality of nodes; and 
 characterizing, by the controller, traffic through individual nodes of the network according to the post-activity data. 
   
     
     
         12 . The method of  claim 11 , further comprising:
 generating, by the first endpoint, the representation of the traffic data for each tuple of the plurality of tuples by summarizing data for a plurality of data packets transferred between the each endpoint and the different endpoint of the each tuple.   
     
     
         13 . The method of  claim 11 , further comprising:
 generating, by the first endpoint, the representation of the traffic data for each tuple of the plurality of tuples by summarizing data for a plurality of data packets transferred between the each endpoint and the different endpoint of the each tuple over a plurality of different socket connections.   
     
     
         14 . The method of  claim 11 , further comprising:
 generating, by the first endpoint, the representation of the traffic data for each tuple of the plurality of tuples by summarizing data for a plurality of data packets transferred between the each endpoint and the different endpoint of each tuple of the plurality of tuples over two or more socket connections during a group time period.   
     
     
         15 . The method of  claim 11 , further comprising:
 transferring, by the first endpoint, the data packets to the other endpoints of the plurality of endpoints through the network using a connection-based packet switching approach.   
     
     
         16 . The method of  claim 15 , further comprising:
 routing, by one or more of the plurality of nodes of the network, the data packets according to a plurality of enforced paths, the topology information including a representation of the plurality of enforced paths.   
     
     
         17 . The method of  claim 11 , further comprising, for each tuple of the plurality of tuples, generating, by the first endpoint, the representation of the traffic data by:
 generating a first representation of a first type of data flow in the traffic data; and   generating a second representation of a second type of data flow in the traffic data, the second type of data flow being of longer duration than the first type of data flow;   wherein each endpoint of the plurality of endpoints is further programmed to transfer both the first representation and the second representation to the controller.   
     
     
         18 . The method of  claim 11 , further comprising:
 for a portion of the tuples of the plurality of tuples, including, by the first endpoint, a path through the network for each tuple of the portion in the representation of the traffic data;   characterizing, by the controller, traffic through nodes of the network according to both the path through the network included in the representation of the traffic data for the first endpoint and the topology information.   
     
     
         19 . The method of  claim 11 , wherein each node of the plurality of nodes operates as at least one of a network switch and a router. 
     
     
         20 . The method of  claim 11 , wherein each endpoint of the plurality of endpoints comprises a computer system including a memory device, a processing device coupled to the memory device, and a network interface controller (NIC) coupled to the processing device.