Streaming Method and System for Processing Network Metadata

Abstract

An improved method and system for processing network metadata is described. Network metadata may be processed by dynamically instantiated executable software modules which make policy-based decisions about the character of the network metadata and about presentation of the network metadata to consumers of the information carried by the network metadata. The network metadata may be type classified and each subclass within a type may be mapped to a definition by a unique fingerprint value. The fingerprint value may be used for matching the network metadata subclasses against relevant policies and transformation rules. For template-based network metadata such as NetFlow v9, an embodiment of the invention can constantly monitor network traffic for unknown templates, capture template definitions, and informs administrators about templates for which custom policies and conversion rules do not exist. Conversion modules can efficiently convert selected types and/or subclasses of network metadata into alternative metadata formats.

What is claimed is: 
     
         1 . A method of improved management of a software-defined network, said network including a network controller and transmitting network traffic using one or more network protocols, the network including devices at least some of which receive network traffic through an ingress interface and transmit network traffic through an egress interface, the method comprising the steps of:
 receiving network metadata from a plurality of sources in a data processing system, in at least one data format;   processing said network metadata while said network metadata is in transition on said network between a network device that generated said network metadata and a device that is able to store said network metadata to extract useful information therefrom; and   determining as a result of said metadata processing step, information relating to applications operating on said network; and   using said applications information to enable said network controller to perform more efficient management of said software-defined network.   
     
     
         2 . The method as set forth in  claim 1 , further comprising the steps of:
 determining as a result of said metadata processing step, information relating to the users present on said network; and   using said user information to enable said network controller to perform more efficient management of said software-defined network.   
     
     
         3 . A method of improved management of a cloud-based virtual computing environment, said environment including a cloud operating system and a cloud environment controller and transmitting network traffic using one or more network protocols, the network including devices at least some of which receive network traffic through an ingress interface and transmit network traffic through an egress interface, the method comprising the steps of:
 receiving network metadata from a plurality of sources in said cloud-based virtual computing environment, in at least one data format;   processing said network metadata while said network metadata is in transition in said environment between a network device that generated said network metadata and a device that is able to store said network metadata to extract useful information therefrom; and   determining as a result of said metadata processing step, information relating to applications operating in said environment; and   using said applications information to enable said cloud environment controller to perform more efficient management of said cloud-based virtual computing environment.   
     
     
         4 . The method as set forth in  claim 3 , further comprising the steps of:
 determining as a result of said metadata processing step, information relating to the users present in said environment; and   using said user information to enable said cloud environment controller to perform more efficient management of said cloud-based virtual computing environment.   
     
     
         5 . A method of providing on-demand access to network metadata relating to an identified potentially security-related network event, in a network in which devices transmit network traffic using one or more network protocols, the network including devices at least some of which receive network traffic through an ingress interface and transmit network traffic through an egress interface, the method comprising the steps of:
 processing network metadata in a streaming fashion according to a configured collection of network metadata processing policies;   retaining a time-indexed set of network metadata in a fast-access storage mechanism for a defined period of time;   identifying a potentially security-related network event; and   providing from said time-indexed set a collection of network metadata related in time to said identified potentially security-related network event; and   performing analysis to correlate said collection of network metadata with said identified potentially security-related network event to further characterize said identified potentially security-related network event.   
     
     
         6 . The method as set forth in  claim 5 , further comprising the step of removing selected network metadata from said fast-access storage mechanism to facilitate the arrival of new network metadata thereto. 
     
     
         7 . A method of detecting botnet slaves on a network-connected device comprising:
 applying an inline cluster analysis algorithm to classify outbound traffic on a network;   said cluster analysis algorithm taking into consideration frequency of communications to the network hosts at identifiable geographic locations and data communication patterns such as, without limitation, application type, flow rate, the number of packets per flow, average packets size in each flow, and traffic rate;   based upon said applying step, identifying outbound traffic on said network that is not part of the general pattern of traffic on said network; and   communicating an alert in the event of outbound traffic on said network that is not part of the general pattern of traffic on said network.   
     
     
         8 . A system for improved management of a software-defined network, said network including a network controller and transmitting network traffic using one or more network protocols, the network including devices at least some of which receive network traffic through an ingress interface and transmit network traffic through an egress interface and generate network metadata relating to said network traffic, said management system comprising:
 at least one ingress interface for receiving network metadata from a plurality of sources in a software-defined network, in at least one data format;   a processing engine for processing said network metadata while said network metadata is in transition on said network between a network device that generated said network metadata and a device that is able to store said network metadata to extract useful information therefrom;   said processing engine determining information relating to applications operating on said network and using said applications information to enable said network controller to perform more efficient management of said software-defined network.   
     
     
         9 . The management system as set forth in  claim 8 , wherein said processing engine determines as a result of said metadata processing step, information relating to the users present on said network; and uses said user information to enable said network controller to perform more efficient management of said software-defined network. 
     
     
         10 . A system for improved management of a cloud-based virtual computing environment, said environment including a cloud operating system and a cloud environment controller that transmits network traffic using one or more network protocols, the network including devices at least some of which receive network traffic through an ingress interface and transmit network traffic through an egress interface, the management system further comprising:
 an interface for receiving network metadata from a plurality of sources in said cloud-based virtual computing environment, in at least one data format;   a processing engine for processing said network metadata while said network metadata is in transition in said environment between a network device that generated said network metadata and a device that is able to store said network metadata to extract useful information therefrom;   said processing engine determining as a result of said metadata processing step, information relating to applications operating in said environment; and using said applications information to enable said cloud environment controller to perform more efficient management of said cloud-based virtual computing environment.   
     
     
         11 . The management system as set forth in  claim 10 , wherein said processing engine determines as a result of said metadata processing step, information relating to the users present in said environment; and uses said user information to enable said cloud environment controller to perform more efficient management of said cloud-based virtual computing environment. 
     
     
         12 . A system for providing on-demand access to network metadata relating to an identified potentially security-related network event, in a network in which devices transmit network traffic using one or more network protocols, the network including devices at least some of which receive network traffic through an ingress interface and transmit network traffic through an egress interface, the system comprising:
 a processing engine for processing network metadata in a streaming fashion according to a configured collection of network metadata processing policies;   a fast-access storage mechanism for retaining a time-indexed set of network metadata for a defined period of time;   said processing engine identifying a potentially security-related network event and providing from said time-indexed set a collection of network metadata related in time to said identified potentially security-related network event; and   an analysis engine for performing analysis to correlate said collection of network metadata with said identified potentially security-related network event to further characterize said identified potentially security-related network event.   
     
     
         13 . The system as set forth in  claim 12 , further comprising a memory management engine for removing selected network metadata from said fast-access storage mechanism to facilitate the arrival of new network metadata thereto. 
     
     
         14 . A system for detecting botnet slaves on a network-connected device comprising:
 a processing engine for applying an inline cluster analysis algorithm to classify outbound traffic on a network;   said cluster analysis algorithm taking into consideration frequency of communications to the network hosts at identifiable geographic locations and data communication patterns such as, without limitation, application type, flow rate, the number of packets per flow, average packets size in each flow, and traffic rate;   an analysis engine that, based upon the results of said cluster analysis algorithm, identifies outbound traffic on said network that is not part of the general pattern of traffic on said network; and   an alert generation engine for communicating an alert in the event of outbound traffic on said network that is not part of the general pattern of traffic on said network.