System and method for managing telemetry data and agents in a telemetry system

Abstract

A system and method include determining, by a telemetry control system of a telemetry system that an agent associated with the telemetry control system terminated during operation. The agent collects telemetry data from data sources associated with the telemetry system. The system and method also include determining that a number of times the agent has terminated is greater than a predetermined threshold, restarting the agent after a first predetermined delay in response to exceeding the predetermined threshold, and determining that the agent terminated again within a predetermined time period upon restarting. The system and method further include updating a configuration file of the agent in response to the termination within the predetermined time period and restarting the agent with the updated configuration file. The updating is based upon an agent termination record of the agent.

What is claimed is: 
     
         1 . A non-transitory computer-readable media comprising computer-readable instructions stored thereon that when executed by a processor causes the processor to:
 monitor a resource limit of a resource allocated to a telemetry agent operating in a telemetry system, wherein the telemetry agent collects and transmits telemetry data, and wherein the processor is associated with the telemetry agent;   troubleshoot the telemetry agent upon determining that a resource utilization of the resource by the telemetry agent has exceeded the resource limit; and   terminate the telemetry agent upon determining that the telemetry agent continues to exceed the resource limit upon troubleshooting.   
     
     
         2 . The non-transitory computer-readable media of  claim 1 , wherein the resource limit comprises a soft limit and a hard limit, and wherein troubleshooting the telemetry agent comprises:
 performing a first troubleshooting action upon the telemetry agent exceeding the soft limit but not the hard limit; and   performing a second troubleshooting action upon the telemetry agent exceeding the hard limit.   
     
     
         3 . The non-transitory computer-readable media of  claim 2 , wherein the processor further comprises computer-readable instructions to terminate the telemetry agent upon determining that the telemetry agent continues to exceed the hard limit upon performing the second troubleshooting action. 
     
     
         4 . The non-transitory computer-readable media of  claim 1 , wherein the processor further comprises computer-readable instructions to:
 perform a troubleshooting action upon the telemetry agent exceeding a soft limit of the resource limit; and   upon determining that the telemetry agent continues to exceed the soft limit upon performing the troubleshooting action, continue to monitor the telemetry agent for the telemetry agent to exceed a hard limit of the resource limit.   
     
     
         5 . The non-transitory computer-readable media of  claim 1 , wherein the processor further comprises computer-readable instructions to:
 perform a troubleshooting action upon the telemetry agent exceeding both a soft limit and a hard limit of the resource limit; and   terminate the telemetry agent upon determining that the telemetry agent continues to exceed the hard limit upon performing the troubleshooting action.   
     
     
         6 . The non-transitory computer-readable media of  claim 1 , wherein the telemetry agent is one of a plurality of telemetry agents, and wherein each of the plurality of telemetry agents is configured to collect a specific type of the telemetry data. 
     
     
         7 . The non-transitory computer-readable media of  claim 1 , wherein the processor further comprises computer-readable instructions to:
 temporarily store the telemetry data collected within a first pre-determined period of time; and   transmit the stored telemetry data upon expiration of a second pre-determined period of time.   
     
     
         8 . The non-transitory computer-readable media of  claim 7 , wherein the processor further comprises computer-readable instructions to combine the telemetry data collected within the first pre-determined period of time before transmission. 
     
     
         9 . The non-transitory computer-readable media of  claim 1 , wherein the telemetry system is configured to:
 update a configuration of the telemetry agent upon the termination of the telemetry agent; and   restart the telemetry agent upon updating the configuration.   
     
     
         10 . The non-transitory computer-readable media of  claim 1 , wherein the processor further comprises computer-readable instructions to send an exit code to the telemetry system upon termination, and wherein the exit codes identifies a cause of the termination. 
     
     
         11 . A method comprising:
 monitoring, by a processor of a telemetry agent operating in a telemetry system, a resource limit of a resource allocated to the telemetry agent that collects and transmits telemetry data;   troubleshooting, by the processor, the telemetry agent upon determining that a resource utilization of the resource by the telemetry agent has exceeded the resource limit; and   terminating, by the processor, the telemetry agent upon determining that the telemetry agent continues to exceed the resource limit upon troubleshooting.   
     
     
         12 . The method of  claim 11 , further comprising:
 determining, by the processor, that the telemetry agent has exceeded a soft limit of the resource limit but not a hard limit of the resource limit; and   performing, by the processor, a troubleshooting action on the telemetry agent upon the telemetry agent exceeding the soft limit.   
     
     
         13 . The method of  claim 11 , further comprising:
 determining, by the processor, that the telemetry agent has exceeded both a soft limit and a hard limit of the resource limit; and   terminating, by the processor, the telemetry agent upon determining that the telemetry agent continues to exceed the hard limit upon performing a troubleshooting action on the telemetry agent.   
     
     
         14 . The method of  claim 11 , further comprising sending, by the processor, an exit code to the telemetry system upon terminating the telemetry agent. 
     
     
         15 . The method of  claim 11 , further comprising:
 updating, by the telemetry system, a configuration of the telemetry agent upon receiving indication of the termination of the telemetry agent; and   restarting, by the telemetry agent, the telemetry agent upon updating the configuration.   
     
     
         16 . The method of  claim 11 , further comprising:
 receiving, by the telemetry system, the telemetry data from the telemetry agent; and   appending, by the telemetry system, the telemetry data to a commit log.   
     
     
         17 . The method of  claim 16 , further comprising:
 deleting, by the telemetry system, the telemetry data from the commit log upon successful transmission of the telemetry data.   
     
     
         18 . A system comprising:
 a memory to store telemetry data and computer-readable instructions;   a processor of a telemetry agent in a telemetry system, wherein the processor executes the computer-readable instructions to:
 monitor a resource limit of a resource allocated to the telemetry agent, wherein the telemetry agent collects and transmits the telemetry data; 
 troubleshoot the telemetry agent upon determining that a resource utilization of the resource by the telemetry agent has exceeded the resource limit; and 
 terminate the telemetry agent upon determining that the telemetry agent continues to exceed the resource limit upon troubleshooting. 
   
     
     
         19 . The system of  claim 18 , wherein the resource limit comprises a soft limit and a hard limit, and wherein the processor executes the computer-readable instructions to terminate the telemetry agent upon the telemetry agent exceeding both the hard limit and the soft limit, but not upon the telemetry agent exceeding the soft limit only. 
     
     
         20 . The system of  claim 18 , wherein telemetry agent is configured to collect the telemetry data from at least one data source in a virtual computing system. 
     
     
         21 . The system of  claim 18 , wherein the processor executes the computer-readable instructions to:
 temporarily store the telemetry data collected within a first pre-determined period of time; and   transmit the stored telemetry data upon expiration of a second pre-determined period of time.   
     
     
         22 . The system of  claim 18 , wherein the processor executes the computer-readable instructions to determine that the telemetry agent has exceeded a soft limit of the resource limit but not a hard limit of the resource limit and to perform a troubleshooting action on the telemetry agent upon the telemetry agent exceeding the soft limit. 
     
     
         23 . The system of  claim 18 , wherein the processor executes the computer-readable instructions to send an exit code to the telemetry system upon terminating the telemetry agent. 
     
     
         24 . The system of  claim 18 , wherein the processor executes the computer-readable instructions to:
 update a configuration of the telemetry agent upon receiving indication of the termination of the telemetry agent; and
 restart the telemetry agent upon updating the configuration.