Instrumentation trace capture technique

Abstract

An instrumentation trace capture technique enables software developers to monitor, diagnose and solve errors associated with application development and production. A client library of an investigative platform is loaded in a user application executing on a virtual machine instance of a virtualized computing environment. The client library interacts with an agent of the platform to instrument executable code of the user application and, to that end, loads a capture configuration that specifies, inter alia, methods and associated arguments, variables and data structures (values), to instrument. The client library inspects the executable code to determine portions of the code to instrument based on the capture configuration, which describes a degree of fidelity (e.g., a frequency) of the executable code and data to trace at runtime. Capture points of the runtime application are implemented as callbacks to the client library, which are registered with a runtime system executing the user application.

What is claimed is: 
     
         1 . A method comprising:
 instrumenting executable code of an application to record information of the application executing on a computer node having a memory, wherein the executable code is instrumented in accordance with a capture configuration determining a fidelity of the recorded information;   generating a capture filter defining the information to capture from the application execution based on event-driven capture intervals established as callbacks into a process of the application;   recording the application execution during the capture interval according to the capture filter, wherein the recording includes information having (i) high-level programming text processed by a runtime system within which the application executes and (ii) symbols, values and associated arguments of methods of the application;   compressing the recording; and   is storing the recording via an agent executing on a same operating system as the application.   
     
     
         2 . The method of  claim 1  wherein recording the application execution during the capture interval further comprises:
 searching the capture filter to determine a presence of one or more identifiers of the information to capture from the application execution; and 
 in response to determining the presence of an identifier of the information to capture, recording the application execution. 
 
     
     
         3 . The method of  claim 2  wherein recording the application execution during the capture interval further comprises:
 querying the runtime system via an application programming interface to determine a trace of the methods and associated arguments of the methods for the application invoked during the capture interval. 
 
     
     
         4 . The method of  claim 1  further comprising:
 in response to an exception raised during the capture interval, increasing the fidelity of the recorded application execution. 
 
     
     
         5 . The method of  claim 1  wherein recording the application execution during the capture interval further comprises:
 determining whether an expression stored in the capture configuration matches a path of a data structure defined in the executable code; and 
 in response to determining that the expression stored in the capture configuration matches the path, recording a runtime value of the data structure. 
 
     
     
         6 . The method of  claim 1  wherein recording the application execution during the capture interval further comprises:
 capturing runtime values of a data structure by walking elements of the data structure in the memory according to a maximum nested depth stored in the capture configuration. 
 
     
     
         7 . The method of  claim 6 , further comprising:
 dynamically adjusting the maximum nested depth stored in the capture configuration based on one of a type of a first element of the data structure and runtime value of the first element of the data structure.   
     
     
         8 . The method of  claim 1  wherein recording the application execution during the capture interval further comprises:
 determining whether an expression stored in the capture configuration matches a type of an element of a data structure defined in the executable code; and 
 in response to determining that the expression stored in the capture configuration matches the type of the element, capturing a value of the element of the structure. 
 
     
     
         9 . The method of  claim 1  wherein the recorded information includes one or more traces of the application method executed during the execution of the application. 
     
     
         10 . The method of  claim 1  wherein the recorded information includes one or more of (i) high-level programming text processed by a runtime system executing the application and (ii) symbols as labels representing one or more of methods, variables, data and state of the executable code. 
     
     
         11 . A non-transitory computer readable medium including program instructions for execution on one or more processors, the program instructions configured to:
 instrument executable code of an application to record information of the application executing on a computer node having a memory, wherein s the executable code is instrumented in accordance with a capture configuration determining a fidelity of the recorded information;   generate a capture filter defining the information to capture from the application execution based on even-driven capture intervals established as callbacks into a process of the application;   record the application execution during the capture interval according to the capture filter, wherein the recording includes information having (i) high-level programming text processed by a runtime system within which the application executes and (ii) symbols, values, and associated arguments of methods of the application;   compress the recording; and   store the recording via an agent executing on a same operating system as the application.   
     
     
         12 . The non-transitory computer readable medium of  claim 11 , wherein the program instructions configured to record the application execution during the capture interval are further configured to:
 search the capture filter to determine a presence of one or more identifiers of the information to capture from the application execution; and   in response to determining the presence of an identifier of the information to capture, record the application execution.   
     
     
         13 . The non-transitory computer readable medium of  claim 12 , wherein the program instructions configured to record the application execution during the capture interval are further configured to:
 query the runtime system via an application programming interface to determine a trace of the methods and associated arguments of the methods for the application invoked during the capture interval.   
     
     
         14 . The non-transitory computer readable medium of  claim 11 , wherein the program instructions configured to record the application execution during the capture interval are further configured to:
 in response to an exception raised during the capture interval, increase the fidelity of the recorded application execution.   
     
     
         15 . The non-transitory computer readable medium of  claim 11 , wherein the program instructions configured to record the application execution during the capture interval are further configured to:
 determine whether an expression stored in the capture configuration matches a path of a data structure defined in the executable code; and   in response to determining that the expression stored in the capture configuration matches the path, capture a runtime value of the data structure.   
     
     
         16 . The non-transitory computer readable medium of  claim 11 , wherein the program instructions configured to record the application execution during the capture interval are further configured to:
 capture runtime values of a data structure by walking elements of the data structure in the memory according to a maximum nested depth stored in the capture configuration.   
     
     
         17 . The non-transitory computer readable medium of  claim 11 , wherein the program instructions configured to record the application execution during the capture interval are further configured to:
 determine whether an expression stored in the capture configuration matches a type of an element of a data structure defined in the executable code; and   in response to determining that the expression stored in the capture configuration matches the type of the element, record a value of the element of the structure.   
     
     
         18 . The non-transitory computer readable medium of  claim 16  wherein the recorded information includes one or more of (i) high-level programming text processed by a runtime system executing the application and (ii) symbols as labels representing one or more of methods, variables, data and state of the executable code. 
     
     
         19 . The non-transitory computer readable medium of  claim 11 , wherein the program instructions configured to compress the recording are further configured to use two-stages of dictionary-based compression for the recorded symbols, values, and associated arguments of the methods, wherein a first stage of the dictionary-based compression is performed in the client library, and a second stage of the dictionary compression is performed in the agent process. 
     
     
         20 . A system comprising:
 a node including a processor and a memory, the memory including an application having executable code linked to a client library with program instructions configured to,
 instrument the executable code to record information of the application executing on the node according to a capture configuration determining a fidelity of the recorded information; 
 generate a capture filter defining the information to capture from the application execution based on even-driven capture intervals established as callbacks into a process of the application; 
 record the application execution during the capture interval according to the capture filter, wherein the recording includes information having (i) high-level programming text processed by a runtime system within which the application executes and (ii) symbols, values, and associated arguments of methods of the application; 
 compress the recording; and 
 store the recording via an agent executing on a same operating system as the application.