Real-time network monitoring and subscriber identification with an on-demand appliance

Abstract

A system and a method are disclosed for selectively monitoring traffic in a service provider network. The system receives a notice for a beginning of a network data flow, which responds to a request from a user device for content at an origin server. The system then determines whether to monitor the data flow from the origin server to the user device. If so determined, the system collects statistic information of the data flow and stores the statistic information to a flow record in a database. The system also maps the flow record to a subscriber of the service provider network by analyzing the statistic information of the data flow and estimates bandwidth provided to the data flow by the service provider's network based on the analysis of the statistic information of the data flow.

What is claimed is: 
     
       1. A method for selectively monitoring traffic in a service provider network, the method comprising:
 receiving a notice for a beginning of a network data flow between an origin server and a user device, the data flow for delivering media content from the origin server to the user device in response to a request for the media content from the user device; 
 determining whether to monitor the data flow between the origin server and the user device; 
 collecting, responsive to a determination to monitor the data flow, statistic information corresponding to the data flow; 
 storing the statistic information collected for the data flow to a flow record in a database; 
 mapping the flow record in the database to a subscriber of the service provider network based on an analysis of the statistic information collected for the data flow; 
 estimating bandwidth to be consumed on the service provider network by the data flow between the user device and the origin server based on the statistic information collected for the data flow; 
 summing the estimated bandwidth of the data flow and estimated bandwidths of at least one other monitored data flow in the service provider network; 
 determining a congestion level of the service provider network based on a comparison of the sum of the estimated bandwidths against a capacity of the service provider network; and 
 determining whether to optimize the data flow based on the congestion level of the service provider network. 
 
     
     
       2. The method of  claim 1 , wherein determining whether to monitor the data flow comprises determining whether the media content in the data flow indicates transfer of video and/or images larger than a predetermined size. 
     
     
       3. The method of  claim 1 , wherein the statistical information collected for the data flow includes at least one of throughput, latency, size of the media content to be delivered, and duration of the flow. 
     
     
       4. The method of  claim 1 , wherein the flow record comprises at least one of: user device source IP address, subscriber identifier, estimated bandwidth, cookie hashes, flow identifier, and transmitted and received bytes, and wherein the flow record is updated during a lifespan of the data flow. 
     
     
       5. The method of  claim 1 , wherein estimating the bandwidth to be consumed on the service provider network by the data flow comprises:
 receiving an update on transmitted or received bytes in the flow record; and 
 calculating the estimated bandwidth by dividing a difference between the transmitted or received bytes since a last update by an interval of time since the last update. 
 
     
     
       6. The method of  claim 1 , wherein determining whether to optimize the data flow based on the congestion level of the service provider network comprises determining to optimize the data flow if the congestion level of the service provider network exceeds the capacity of the service provider network. 
     
     
       7. The method of  claim 1 , wherein determining whether to optimize the data flow based on the congestion level of the service provider network comprises determining not to optimize the data flow if the congestion level of the service provider network does not exceed the capacity of the service provider network. 
     
     
       8. The method of  claim 1 , wherein the determination of whether to optimize the data flow is further based on a size of the media content to be delivered in the data flow. 
     
     
       9. The method of  claim 1 , further comprising redirecting, in response to determining to optimize the data flow, the request for the media content from the requesting user device to a video optimizer, the video optimizer retrieving the media content from the origin server, optimizing the media content for the user device, and transmitting the optimized media content to the user device. 
     
     
       10. The method of  claim 1 , wherein aggregating flow records corresponding to the subscriber in the database of the service provider network is based on matching at least one of: user device source IP address and MAC address, TCP port number, and hashes of a cookie associated with the data flows. 
     
     
       11. A non-transitory computer-readable storage medium storing executable computer program instructions for selectively monitoring traffic in a service provider network, the computer program instructions comprising instructions that cause a processor to:
 receive a notice for a beginning of a network data flow between an origin server and a user device, the data flow for delivering media content from the origin server to the user device in response to a request for the media content from the user device; 
 determine whether to monitor the data flow between the origin server and the user device; 
 collect, in responsive to a determination to monitor the data flow, statistic information corresponding to the data flow; 
 store the statistic information collected for the data flow to a flow record in a database; 
 map the flow record in the database to a subscriber of the service provider network based on an analysis of the statistic information collected for the data flow; 
 estimate bandwidth to be consumed on the service provider network by the data flow between the user device and the origin server based on the statistic information collected for the data flow; 
 sum the estimated bandwidth of the data flow and estimated bandwidths of at least one other monitored data flow in the service provider network; 
 determine a congestion level of the service provider network based on a comparison of the sum of the estimated bandwidths against a capacity of the service provider network; and 
 determine whether to optimize the data flow based on the congestion level of the service provider network. 
 
     
     
       12. The non-transitory computer-readable storage medium of  claim 11 , wherein the instructions that cause the processor to determine whether to monitor the data flow comprises instructions that cause the processor to determine whether the media content in the data flow indicates transfer of video and/or images larger than a predetermined size. 
     
     
       13. The non-transitory computer-readable storage medium of  claim 11 , wherein the statistical information collected for the data flow includes at least one of throughput, latency, size of the media content to be delivered, and duration of the flow. 
     
     
       14. The non-transitory computer-readable storage medium of  claim 11 , wherein the flow record comprises at least one of: user device source IP address, subscriber identifier, estimated bandwidth, cookie hashes, flow identifier, and transmitted and received bytes, and wherein the flow record is updated during a lifespan of the data flow. 
     
     
       15. The non-transitory computer-readable storage medium of  claim 11 , wherein the instructions that cause the processor to estimate the bandwidth to be consumed on the service provider network by the data flow comprises instructions that cause the processor to:
 receive an update on transmitted or received bytes in the flow record; and 
 calculate the estimated bandwidth by dividing a difference between the transmitted or received bytes since a last update by an interval of time since the last update. 
 
     
     
       16. The non-transitory computer-readable storage medium of  claim 11 , wherein the instructions that cause the processor to determine whether to optimize the data flow based on the congestion level of the service provider network comprises instructions that cause the processor to determine to optimize the data flow if the congestion level of the service provider network exceeds the capacity of the service provider network. 
     
     
       17. The non-transitory computer-readable storage medium of  claim 11 , wherein the instructions that cause the processor to determine whether to optimize the data flow based on the congestion level of the service provider network comprises instructions that cause the processor to determine not to optimize the data flow if the congestion level of the service provider network does not exceed the capacity of the service provider network. 
     
     
       18. The non-transitory computer-readable storage medium of  claim 11 , the computer program instructions further comprising instructions that cause the processor to redirect, in response to determining to optimize the data flow, the request for the media content from the requesting user device to a video optimizer, the video optimizer retrieving the media content from the origin server, optimizing the media content for the user device, and transmitting the optimized media content to the user device. 
     
     
       19. The non-transitory computer-readable storage medium of  claim 11 , wherein the instructions that cause the processor to aggregate flow records corresponding to the subscriber in the database of the service provider network comprises instructions to match to at least one of: user device source IP address and MAC address, TCP port number, and hashes of a cookie associated with the data flows. 
     
     
       20. A system for selectively monitoring traffic in a service provider network, comprising:
 a processor; and 
 a non-transitory computer-readable storage medium storing executable computer program instructions that when executed by the processor:
 receive a notice for a beginning of a network data flow between an origin server and a user device, the data flow for delivering media content from the origin server to the user device in response to a request for the media content from the user device; 
 determine whether to monitor the data flow between the origin server and the user device; 
 collect statistic information corresponding to the data flow in response to a determination to monitor the data flow; 
 store the statistic information collected for the data flow to a flow record in a database; 
 map the flow record in the database to a subscriber of the service provider network based on an analysis of the statistic information collected for the data flow; and 
 estimate bandwidth to be consumed on the server provider network by the data flow between the user device and the origin server based on the statistic information collected for the data flow; 
 sum the estimated bandwidth of the data flow and estimated bandwidths of at least one other monitored data flow in the service provider network; 
 determine a congestion level of the service provider network based on a comparison of the sum of the estimated bandwidths against a capacity of the service provider network; and 
 determine whether to optimize the data flow based on the congestion level of the service provider network. 
 
 
     
     
       21. The system of  claim 20 , wherein the system collects statistic information of the data flow as an inline network element that receives and forwards the data flow. 
     
     
       22. The system of  claim 20 , wherein the system receives the data flow from an inline network element in response to an initial determination to monitor the data flow based on initial statistic information collected for the data flow, the inline network element forwarding the data flow for processing in response to the initial determination to monitor the data flow. 
     
     
       23. The system of  claim 22 , wherein determining whether to monitor the data flow between the origin server and the user device further comprises, in response to a determination to not monitor the data flow, transmitting an instruction causing the inline network element to cease forwarding of the data flow through the system. 
     
     
       24. The system of  claim 20 , wherein estimating the bandwidth to be consumed on the service provider network by the data flow comprises:
 receiving an update on transmitted or received bytes in the flow record; and 
 calculating the estimated bandwidth by dividing a difference between the transmitted or received bytes since a last update by an interval of time since the last update. 
 
     
     
       25. The system of  claim 20 , wherein determining whether to optimize the data flow based on the congestion level of the service provider network comprises determining to optimize the data flow if the congestion level of the service provider network exceeds the capacity of the service provider network. 
     
     
       26. The system of  claim 20 , wherein determining whether to optimize the data flow based on the congestion level of the service provider network comprises determining not to optimize the data flow if the congestion level of the service provider network does not exceed the capacity of the service provider network. 
     
     
       27. The system of  claim 20 , the computer program instructions further comprising instructions for redirecting, in response to determining to optimize the data flow, the request for the media content from the requesting user device to a video optimizer, the video optimizer retrieving the media content from the origin server, optimizing the media content for the user device, and transmitting the optimized media content to the user device.