Systems and methods for extracting structured application data from a communications link

Abstract

Systems and methods for generating a semantic description of operations between network agents. In an embodiment, packet-level traffic between two or more network agents is captured. The packet-level traffic is bundled into one or more messages, wherein each message comprises one or more elements. For each of the messages, the elements of the message are matched to one or more attributes, and the message is decoded into message data based on the matched attributes. The message data is then used to generate a semantic description of operations between the network agents.

What is claimed is: 
     
         1 . A method for generating a semantic description of operations between network agents, the method comprising, using at least one hardware processor:
 capturing packet-level traffic between a first network agent and a second network agent;   bundling the packet-level traffic into one or more messages, wherein each of the one or more messages comprises one or more elements;   for each of the one or more messages,
 matching one or more of the one or more elements of the message to one or more attributes, and 
 decoding the message into message data based on the matched one or more attributes; and 
   generating a semantic description of operations between the first network agent and the second network agent based on the message data.   
     
     
         2 . The method of  claim 1 , wherein capturing packet-level traffic between a first network agent and a second network agent comprises receiving copies of packets from a switch positioned on a communicative path between the first network agent and the second network agent. 
     
     
         3 . The method of  claim 2 , wherein the switch is a virtual switch. 
     
     
         4 . The method of  claim 1 , wherein capturing packet-level traffic between a first network agent and a second network agent comprises receiving copies of packets from a network tap positioned on a communicative path between the first network agent and the second network agent. 
     
     
         5 . The method of  claim 4 , wherein the network tap is a virtual network tap. 
     
     
         6 . The method of  claim 1 , wherein bundling the packet-level traffic into one or more messages comprises, iteratively, for each of one more sessions:
 receiving a packet transmitted between the first network agent and the second network agent;   determining whether a boundary condition has occurred, wherein the boundary condition indicates an end of a message;   if it is determined that the boundary condition has not occurred, queuing the received packet in memory; and,   if it is determined that the boundary condition has occurred, bundling one or more packets queued in the memory into a message.   
     
     
         7 . The method of  claim 6 , wherein determining whether a boundary condition has occurred comprises one or more of:
 detecting control information indicating an end of a message;   detecting a change in direction of communication between the first network agent and the second network agent based on the received packet; and   detecting that a timeout has occurred since reception of a packet.   
     
     
         8 . The method of  claim 1 , wherein matching one or more of the one or more elements of the message to one or more attributes comprises, for each template in a set of one or more templates:
 selecting the template, wherein the template comprises one or more attributes;   comparing one or more of the one or more attributes of the template to one or more of the one or more elements of the message; and   determining whether the template matches the message based on the comparison.   
     
     
         9 . The method of  claim 8 , further comprising selecting the set of one or more templates from a plurality of templates. 
     
     
         10 . The method of  claim 9 , wherein the set of one or more templates is selected based on startup information for a connection between the first network agent and the second network agent. 
     
     
         11 . The method of  claim 9 , wherein the set of one or more templates is selected based on a stored set of one or more templates used for a prior connection between the first network agent and the second network agent. 
     
     
         12 . The method of  claim 8 , further comprising, for each of the one or more messages:
 selecting one or more templates that match the message from the set of one or more templates; and   decoding the message based on the one or more attributes of the selected one or more templates.   
     
     
         13 . The method of  claim 8 , wherein the set of one or more templates comprises one or more templates corresponding to one or more of Transmission Control Protocol (TCP), Internet Protocol (IP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), Sequenced Packet Exchange protocol (SPX), Address Resolution Protocol (ARP), Transparent Network Substrate protocol (TNS), Tabular Data Stream protocol (TDS), Symmetric Multiprocessing protocol (SMP), Two Task Common protocol (TTC), Network File System protocol (NFS), Apple Filing Protocol (AFP), Server Message Block protocol (SMB), Domain Name System protocol (DNS), Simple Mail Transfer Protocol (SMTP), Internet Message Access Protocol (IMAP), and Post Office Protocol (POP). 
     
     
         14 . The method of  claim 1 , further comprising discarding at least a portion of one or both of the packet-level traffic and message data, based on one or more filters, prior to generating the semantic description of operations. 
     
     
         15 . The method of  claim 1 , wherein the one or more messages comprise one or more of a request message for a remote procedure call (RPC) and a response message for an RPC, and wherein decoding each of the one or more messages into message data comprises extracting one or more of verbs, parameters, and result payloads for an RPC. 
     
     
         16 . The method of  claim 1 , wherein the capturing is performed by a capture module, wherein the bundling, matching, decoding, and generating is performed by an analysis module, and wherein the method further comprises passing the packet-level traffic from the capture module to the analysis module using at least one application programming interfaces (API). 
     
     
         17 . The method of  claim 16 , wherein the capture module and analysis module are executed on separate machines. 
     
     
         18 . The method of  claim 16 , wherein the capture module and analysis module are executed on one machine. 
     
     
         19 . A system for generating a semantic description of operations between network agents, the system comprising:
 at least one hardware processor; and   at least one executable software module that, when executed by the at least one hardware processor,
 captures packet-level traffic between a first network agent and a second network agent, 
 bundles the packet-level traffic into one or more messages, wherein each of the one or more messages comprises one or more elements, 
 for each of the one or more messages,
 matches one or more of the one or more elements of the message to one or more attributes, and 
 decodes the message into message data based on the matched one or more attributes, and 
 
 generates a semantic description of operations between the first network agent and the second network agent based on the message data. 
   
     
     
         20 . The system of  claim 19 , wherein capturing packet-level traffic between a first network agent and a second network agent comprises receiving copies of packets from a switch positioned on a communicative path between the first network agent and the second network agent. 
     
     
         21 . The system of  claim 20 , wherein the switch is a virtual switch. 
     
     
         22 . The system of  claim 19 , wherein capturing packet-level traffic between a first network agent and a second network agent comprises receiving copies of packets from a network tap positioned on a communicative path between the first network agent and the second network agent. 
     
     
         23 . The system of  claim 22 , wherein the network tap is a virtual network tap. 
     
     
         24 . The system of  claim 19 , wherein bundling the packet-level traffic into one or more messages comprises, iteratively, for each of one more sessions:
 receiving a packet transmitted between the first network agent and the second network agent;   determining whether a boundary condition has occurred, wherein the boundary condition indicates an end of a message;   if it is determined that the boundary condition has not occurred, queuing the received packet in memory; and,   if it is determined that the boundary condition has occurred, bundling one or more packets queued in the memory into a message.   
     
     
         25 . The system of  claim 24 , wherein determining whether a boundary condition has occurred comprises one or more of:
 detecting control information indicating an end of a message;   detecting a change in direction of communication between the first network agent and the second network agent based on the received packet; and   detecting that a timeout has occurred since reception of a packet.   
     
     
         26 . The system of  claim 19 , wherein matching one or more of the one or more elements of the message to one or more attributes comprises, for each template in a set of one or more templates:
 selecting the template, wherein the template comprises one or more attributes;   comparing one or more of the one or more attributes of the template to one or more of the one or more elements of the message; and   determining whether the template matches the message based on the comparison.   
     
     
         27 . The system of  claim 26 , wherein the at least one executable software module selects the set of one or more templates from a plurality of templates. 
     
     
         28 . The system of  claim 27 , wherein the set of one or more templates is selected based on startup information for a connection between the first network agent and the second network agent. 
     
     
         29 . The system of  claim 27 , wherein the set of one or more templates is selected based on a stored set of one or more templates used for a prior connection between the first network agent and the second network agent. 
     
     
         30 . The system of  claim 26 , wherein the at least one executable software module, for each of the one or more messages:
 selects one or more templates that match the message from the set of one or more templates; and   decodes the message based on the one or more attributes of the selected one or more templates.   
     
     
         31 . The system of  claim 26 , wherein the set of one or more templates comprises one or more templates corresponding to one or more of Transmission Control Protocol (TCP), Internet Protocol (IP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), Sequenced Packet Exchange protocol (SPX), Address Resolution Protocol (ARP), Transparent Network Substrate protocol (TNS), Tabular Data Stream protocol (TDS), Symmetric Multiprocessing protocol (SMP), Two Task Common protocol (TTC), Network File System protocol (NFS), Apple Filing Protocol (AFP), Server Message Block protocol (SMB), Domain Name System protocol (DNS), Simple Mail Transfer Protocol (SMTP), Internet Message Access Protocol (IMAP), and Post Office Protocol (POP). 
     
     
         32 . The system of  claim 19 , wherein the at least one executable software module discards at least a portion of one or both of the packet-level traffic and message data, based on one or more filters, prior to generating the semantic description of operations. 
     
     
         33 . The system of  claim 19 , wherein the one or more messages comprise one or more of a request message for a remote procedure call (RPC) and a response message for an RPC, and wherein decoding each of the one or more messages into message data comprises extracting one or more of verbs, parameters, and result payloads for an RPC. 
     
     
         34 . The system of  claim 19 , wherein the capturing is performed by a capture module, wherein the bundling, matching, decoding, and generating is performed by an analysis module, and wherein the capture module passes the packet-level traffic to the analysis module using at least one application programming interface (API). 
     
     
         35 . The system of  claim 34 , wherein the system further comprises a first machine and a second machine, and wherein the capture module is hosted on the first machine and the analysis module is hosted on the second machine. 
     
     
         36 . The system of  claim 35 , wherein the system further comprises a single machine that comprises the at least one hardware processor, the capture module, and the analysis module.