Distributed packet deduplication

Abstract

Introduced here are network visibility appliances capable of implementing a distributed deduplication scheme by routing traffic amongst multiple instances of a deduplication program. Data traffic can be forwarded to a pool of multiple network visibility appliances that collectively ensure no duplicate copies of data packets exist in the data traffic. The network visibility appliances can route the traffic to different instances of the deduplication program so that duplicate copies of a data packet are guaranteed to arrive at the same instance of the deduplication program, regardless of which network visibility appliance(s) initially received the duplicate copies of the data packet.

What is claimed is: 
     
         1 . A computer-implemented method comprising:
 receiving virtualized traffic at a first ingress port of a first network visibility appliance;   separating the virtualized traffic into
 a first batch of data packets that share a first characteristic in common, the first batch to be examined by a first instance of a deduplication program that executes in the first network visibility appliance, and 
 a second batch of data packets that share a second characteristic in common, the second batch to be examined by a second instance of the deduplication program that executes in a second network visibility appliance; 
   receiving a third batch of data packets at a second ingress port of the first network visibility appliance;   determining that all data packets in the third batch share the first characteristic in common;   forwarding all data packets of the first batch and all data packets of the third batch to the first instance of the deduplication for examination; and   forwarding all data packets of the second batch to an egress port for transmission to the second network visibility appliance.   
     
     
         2 . The computer-implemented method of  claim 1 , wherein the virtualized traffic is associated with a first virtual machine, and wherein the third batch of data packets is included in virtualized traffic associated with a second virtual machine. 
     
     
         3 . The computer-implemented method of  claim 1 , wherein said separating comprises:
 generating a hash value for each data packet included in the virtualized traffic, thereby producing a plurality of hash values;   accessing a data structure that maps the plurality of hash values amongst a plurality of instances of the deduplication program,
 wherein each hash value is mapped to only a single instance of the deduplication program, and 
 wherein each instance of the deduplication program executes in a different network visibility appliance; 
   determining that each data packet in the first batch of data packets corresponds to an entry in the data structure that specifies the first instance of the deduplication program; and   determining that each data packet in the second batch of data packets corresponds to an entry in the data structure that specifies the second instance of the deduplication program.   
     
     
         4 . The computer-implemented method of  claim 3 , further comprising:
 receiving an indication that the second network visibility appliance is not presently accessible; and   modifying entries in the data structure that correspond to the second batch of data packets to indicate a third instance of the deduplication program,
 wherein the third instance of the deduplication program resides on a third network visibility appliance, and 
 wherein said modifying causes the second batch of data packets to be forwarded to a second egress port for transmission to the third network visibility appliance. 
   
     
     
         5 . A computer-implemented method comprising:
 applying, by a network appliance, a specified transformation function to generate a value for each of a plurality of data packets received at a first ingress port of the network appliance, to produce a plurality of values;   using, by the network appliance, the plurality of values to identify
 a first batch of data packets to be examined by a local instance of a deduplication program that executes in the network appliance, and 
 a second batch of data packets to be examined by a remote instance of the deduplication program that executes in another network appliance; and 
   forwarding, by the network appliance, the second batch of data packets to an egress port for transmission to the other network appliance.   
     
     
         6 . The computer-implemented method of  claim 5 , further comprising:
 causing, by the network appliance, the first batch of data packets to be examined by the local instance of the deduplication program; and   forwarding, by the network appliance, at least a portion of the first batch of data packets to a tool port for transmission to a network tool.   
     
     
         7 . The computer-implemented method of  claim 6 , further comprising:
 receiving, by the network appliance, a third batch of data packets from the other network appliance at a second ingress port;   causing, by the network appliance, the third batch of data packets to be examined by the local instance of the deduplication program; and   forwarding, by the network appliance, at least a portion of the third batch of data packets to the tool port for transmission to the network tool.   
     
     
         8 . The computer-implemented method of  claim 7 , wherein the second ingress port corresponds to one end of a tunnel connected between the network appliance and the other network appliance. 
     
     
         9 . The computer-implemented method of  claim 5 , wherein using the plurality of values to identify the first and second batches of data packets comprises:
 accessing, by the network appliance, a data structure that maps the plurality of values amongst a plurality of instances of the deduplication program,
 wherein each value is mapped to only a single instance of the deduplication program, and 
 wherein each instance of the deduplication program executes in a different network appliance; 
   determining, by the network appliance, that each data packet in the first batch of data packets corresponds to an entry that specifies the local instance of the deduplication program; and   determining, by the network appliance, that each data packet in the second batch of data packets corresponds to an entry that specifies the remote instance of the deduplication program.   
     
     
         10 . The computer-implemented method of  claim 5 , wherein each value is based on a field in a header of the plurality of data packets. 
     
     
         11 . The computer-implemented method of  claim 5 , wherein each value is based on at least one of: Transmission Control Protocol (TCP) sequence number, header length, payload length, type of service, protocol, source address, or destination address. 
     
     
         12 . A computer-implemented method comprising:
 identifying a plurality of network visibility appliances to be included in a distributed packet deduplication scheme;   instantiating a plurality of load balancers associated with the plurality of network visibility appliances;   instantiating a plurality of instances of a deduplication program on the plurality of network visibility appliances; and   establishing a communication channel between a first network visibility appliance of the plurality of network visibility appliances and a second network visibility appliance of the plurality of network visibility appliances.   
     
     
         13 . The computer-implemented method of  claim 12 , wherein each instance of the plurality of instances of the deduplication program and each load balancer of the plurality of load balancers executes in a different network appliance of the plurality of network visibility appliances. 
     
     
         14 . The computer-implemented method of  claim 12 , further comprising:
 configuring the plurality of load balancers to apply an identical transformation function to incoming data packets to the plurality of network visibility appliances.   
     
     
         15 . The computer-implemented method of  claim 12 , further comprising:
 determining a number of network visibility appliances based on an expected volume of traffic,   wherein the plurality of network visibility appliances to be included in the distributed deduplication scheme include the determined number of network visibility appliances.   
     
     
         16 . The computer-implemented method of  claim 15 , wherein the expected volume of traffic is estimated based on a historical volume or a number of virtual machines to be monitored. 
     
     
         17 . The computer-implemented method of  claim 12 , wherein establishing the communication channel includes:
 for each network visibility appliance of the plurality of network visibility appliances, configuring a corresponding list of network ports,   wherein each entry in a list of network ports indicates (1) an instance of the plurality of instances of the deduplication program executing on the network visibility appliance, or (2) a tunnel to another network visibility appliance.   
     
     
         18 . The computer-implemented method of  claim 12 , wherein a load balancer associated with a first network visibility appliance of the plurality of network visibility appliances is configured to:
 generate a value for each data packet included in virtualized traffic received at an ingress port of the first network visibility appliance, thereby producing a plurality of values;   identify a first value corresponding to a first data packet included in the virtualized traffic;   access a data structure that includes an entry for each value, wherein each entry includes a routing instruction that specifies which instance of the plurality of instances of the deduplication program is responsible for examining the corresponding data packet;   determine, based on the first value, that the first data packet is to be examined by a remote instance of the deduplication program that executes in a second network visibility appliance; and   forward the first data packet to an egress port for transmission to the second network visibility appliance.   
     
     
         19 . The computer-implemented method of  claim 18 , wherein the load balancer is further configured to:
 identify a second value corresponding to a second data packet included in the virtualized traffic;   determine, based on the second value, that the second data packet is to be examined by a local instance of the deduplication program that executes in the first network visibility appliance; and   cause the second data packet to be examined by the local instance of the deduplication program.   
     
     
         20 . The computer-implemented method of  claim 12 , wherein the plurality of network visibility appliances include a tool port through which to route traffic to a network tool.