Application regression detection in computing systems

Abstract

Computing systems, devices, and associated methods of detecting application regression in a distributed computing system are disclosed herein. In one embodiment, a method includes receiving data representing telemetry records from one or more hosts of the distributed computing system. At least some of the telemetry records are exception records individually indicating an operation by a user application has failed during execution. The method also includes determining a failure rate of executing the operation by the user application while compensating for a workload of the user application in the distributed computing system. A comparison is performed between the determined failure rate and a threshold. Based on the performed comparison, a regression notification can be generated to indicate that application regression has occurred notwithstanding the workload of the user application in the distributed computing system.

I claim: 
     
       1. A method for detecting application regression in a distributed computing system having multiple hosts interconnected by a computer network, an individual host of the multiple hosts executing a user application to provide a cloud service to tenants, the method comprising:
 receiving, via the computer network, data representing telemetry records from the user application executing via the individual host, wherein at least one telemetry record of the telemetry records is an exception record indicating that an operation by the user application has failed during execution; 
 deriving an exception rate of the operation based on the data representing the telemetry records; 
 compensating the exception rate using a total number of occurrences in which the operation is executed by the user application; 
 determining that application regression has occurred in the user application based on the compensated exception rate; and 
 in response to determining that the application regression has occurred in the user application, generating, in the computer network, a regression notification indicating that the application regression has occurred in the user application. 
 
     
     
       2. The method of  claim 1 , further comprising aggregating the data representing the telemetry records for each of multiple time intervals within a predefined timespan, wherein deriving the exception rate includes deriving an error percentage for each of the multiple time intervals within the predefined timespan using the aggregated data representing the telemetry records for each of the multiple time intervals. 
     
     
       3. The method of  claim 2 , wherein the application regression is determined to have occurred in the user application based on at least one of the error percentages being above an error percentage threshold. 
     
     
       4. The method of  claim 3 , wherein the error percentage threshold is an average value of the error percentages derived for the multiple time intervals. 
     
     
       5. The method of  claim 3 , wherein the error percentage threshold is based on historical error percentages derived for the multiple time intervals. 
     
     
       6. The method of  claim 1 , wherein compensating the exception rate includes:
 determining that the data representing the telemetry records is related to multiple operations; and 
 in response to determining that the data representing the telemetry records is related to the multiple operations, deriving a distinct exception rate related to each of the multiple operations by dividing a number of sessions in which each of the multiple operations has failed by a total number of sessions in which each of the multiple operations is executed. 
 
     
     
       7. The method of  claim 1 , wherein compensating the exception rate includes:
 determining that the data representing the telemetry records is related to a single operation; and 
 in response to determining that the data representing the telemetry records is related to the single operation, determining that the data representing the telemetry records is related to multiple sessions in which the single operation is executed; and 
 in response to determining that the data representing the telemetry records is related to the multiple sessions in which the single operation is executed, deriving an error percentage related to the single operation as the exception rate by dividing a number of sessions in which the single operation has failed by a total number of sessions in which the single operation is executed. 
 
     
     
       8. The method of  claim 1 , wherein compensating the exception rate includes:
 determining that the data representing the telemetry records is related to a single operation; and 
 in response to determining that the data representing the telemetry records is related to the single operation, determining that the data representing the telemetry records is not related to multiple sessions in which the single operation is executed; and 
 in response to determining that the data representing the telemetry records is not related to the multiple sessions in which the single operation is executed, deriving an error percentage related to the single operation as the exception rate by dividing a number of exception records by a total number of the telemetry records. 
 
     
     
       9. The method of  claim 1 , wherein:
 the exception record indicates the operation by the user application has failed during execution after a software update, a software patch, or a configuration change is applied to the user application; and 
 the regression notification indicates that the software update, the software patch, or the configuration change has caused the application regression to occur in the user application. 
 
     
     
       10. The method of  claim 1 , further comprising sending the regression notification to an entity able to perform a remedial action to reverse the application regression. 
     
     
       11. A computing device in a distributed computing system having multiple hosts interconnected by a computer network, an individual host of the multiple hosts executing a user application to provide a cloud service to tenants, the computing device comprising:
 a processor; and 
 a memory operatively coupled to the processor, the memory containing instructions executable by the processor to cause the computing device to:
 perform an analysis of telemetry records received via the computer network from the individual host, wherein at least one telemetry record of the telemetry records is an exception record indicating that an operation by the user application has failed during execution on the individual host; 
 based on the analysis of the telemetry records, derive an exception rate of the operation based on the telemetry records; 
 compensate the exception rate using a total number of occurrences in which the operation is executed by the user application; 
 determine that application regression has occurred in the user application based on the compensated exception rate; and 
 in response to determining that the application regression has occurred in the user application, generate, in the computer network, a regression notification indicating that the application regression has occurred in the user application. 
 
 
     
     
       12. The computing device of  claim 11 , wherein performing the analysis includes:
 determining that the telemetry records are related to multiple operations; and 
 in response to determining that the telemetry records are related to the multiple operations, deriving a distinct error percentage related to each of the multiple operations by dividing a number of sessions in which each of the multiple operations has failed by a total number of sessions in which each of the multiple operations is executed. 
 
     
     
       13. The computing device of  claim 11 , wherein performing the analysis includes:
 determining that the telemetry records are related to a single operation; and 
 in response to determining that the telemetry records are related to the single operation, determining that the exception record is related to multiple sessions in which the single operation is executed; and 
 in response to determining that the exception record is related to the multiple sessions in which the single operation is executed, deriving an error percentage related to the single operation by dividing a number of sessions in which the single operation has failed by a total number of sessions in which the single operation is executed. 
 
     
     
       14. The computing device of  claim 11 , wherein the regression notification indicates that a software update, a software patch, or a configuration change has caused the application regression of the user application. 
     
     
       15. The computing device of  claim 11 , wherein the instructions are further executable by the processor to cause the computing device to send the regression notification to an entity able to perform a remedial action to reverse the application regression. 
     
     
       16. A method for detecting application regression in a distributed computing system having multiple hosts interconnected by a computer network, an individual host of the multiple hosts executing a user application to provide a cloud service to tenants, the method comprising:
 receiving, via the computer network, data representing telemetry records from the individual host, at least one telemetry record of the telemetry records is an exception record indicating that an operation by the user application has failed during execution; 
 determining a failure rate of executing the operation by the user application in the distributed computing system based on the data representing the telemetry records; 
 compensating the failure rate based on a workload of the user application; 
 performing a comparison of the failure rate with a threshold; and 
 based on the comparison of the failure rate with the threshold, generating, in the computer network, a regression notification indicating that application regression has occurred in the user application. 
 
     
     
       17. The method of  claim 16 , further comprising aggregating the data representing the telemetry records for each of multiple time intervals within a timespan, wherein determining the failure rate includes deriving a failure rate for each of the multiple time intervals within the timespan using the aggregated telemetry records for each of the multiple time intervals. 
     
     
       18. The method of  claim 17 ,
 wherein performing the comparison includes determining whether at least one of the derived failure rates is above the threshold. 
 
     
     
       19. The method of  claim 16 , wherein performing the analysis includes:
 determining that the telemetry records are related to multiple operations; and 
 in response to determining that the telemetry records are related to the multiple operations, deriving a distinct failure rate related to each of the multiple operations by dividing a number of sessions in which each of the multiple operations has failed by a total number of sessions in which each of the multiple operations is executed. 
 
     
     
       20. The method of  claim 16 , further comprising sending the regression notification to an entity able to perform a remedial action to reverse the application regression.