Determining the health of an iot application

Abstract

The example embodiments are directed to a system and method for monitoring the health of analytical applications. In certain embodiments, these analytic applications may be a part of a broader IoT solution that may optionally be hosted on a cloud platform. In one example, the method includes receiving an output of an IoT analytic application deployed on a computing platform, selecting at least one performance metric based on a type of the application, calculating a performance of the application based on the selected performance metric, the received output of the application, and an expected output of the application, and in response to detecting the calculated performance of the application is below a predetermined threshold, outputting an alert to a user device. Accordingly, the health of an IoT analytic application can be monitored and an alert can be provided when the application operates below a predetermined threshold.

What is claimed is: 
     
         1 . A computing system comprising:
 a storage configured to store predictions made by an analytic application;   a processor configured to
 select at least one performance metric for evaluating an accuracy of the analytic application based on a type of the analytic application, 
 determine an accuracy value of the predictions made by the analytic application based on the selected performance metric, the predictions made by the analytic application, and an expected predictive output of the analytic application, and 
 determine that a predictive capability of the analytic application has deteriorated based on the determined accuracy value of the predictions; and 
   a network interface configured to transmit an alert to a user device associated with the analytic application.   
     
     
         2 . The computing system of  claim 1 , wherein the processor determines how accurate the predictions made by the analytic application are via execution of the selected at least one performance metric. 
     
     
         3 . The computing system of  claim 1 , wherein the accuracy value of the analytic application identifies how accurate the analytic application is for a predetermined period of time. 
     
     
         4 . The computing system of  claim 1 , wherein the processor is further configured to display, via a user interface, a list of performance metrics associated with the analytic application that are capable of being selected by a user to measure the predictive accuracy of the analytic application. 
     
     
         5 . The computing system of  claim 1 , wherein the at least one performance metric comprises at least one of a receiver operating characteristic (ROC) curve, a confusion matrix, and a lift chart, which compare the predictions made by the analytic application with respect to the expected predictive output of the analytic application. 
     
     
         6 . The computing system of  claim 1 , wherein the at least one performance metric comprises at least one of R squared, mean absolute error (MAE), mean squared error (MSE), and mean square root error (RMSE), which compare the predictions made by the analytic application with respect to the expected predictive output of the analytic application. 
     
     
         7 . The computing system of  claim 1 , wherein the at least one performance metric comprises a confidence interval plot, which compare the predictions made by the analytic application with respect to the expected predictive output of the analytic application. 
     
     
         8 . The computing system of  claim 1 , wherein the processor is further configured to receive new predictions that are made by the analytic application, and determine a new accuracy value of the newly received predictions made by the analytic application based on the selected performance metric, the newly received predictions made by the analytic application, and the expected predictive output of the analytic application. 
     
     
         9 . A method comprising:
 receiving predictions made by an analytic application;   selecting at least one performance metric for evaluating an accuracy of the analytic application based on a type of the analytic application;   determining an accuracy value of the predictions made by the analytic application based on the selected performance metric, the predictions made by the analytic application, and an expected predictive output of the analytic application;   determining that a predictive capability of the analytic application has deteriorated based on the determined accuracy value of the predictions; and   transmitting an alert to a user device associated with the analytic application.   
     
     
         10 . The method of  claim 9 , wherein the determining comprises determining how accurate the predictions made by the analytic application are via execution of the selected at least one performance metric. 
     
     
         11 . The method of  claim 9 , wherein the accuracy value of the analytic application identifies how accurate the analytic application is for a predetermined period of time. 
     
     
         12 . The method of  claim 9 , further comprising displaying, via a user interface, a list of performance metrics associated with the analytic application that are capable of being selected by a user to measure the predictive accuracy of the analytic application. 
     
     
         13 . The method of  claim 9 , wherein the at least one performance metric comprises at least one of a receiver operating characteristic (ROC) curve, a confusion matrix, and a lift chart, which compare the predictions made by the analytic application with respect to the expected predictive output of the analytic application. 
     
     
         14 . The method of  claim 9 , wherein the at least one performance metric comprises at least one of R squared, mean absolute error (MAE), mean squared error (MSE), and mean square root error (RMSE), which compare the predictions made by the analytic application with respect to the expected predictive output of the analytic application. 
     
     
         15 . The method of  claim 9 , wherein the at least one performance metric comprises a confidence interval plot, which compare the predictions made by the analytic application with respect to the expected predictive output of the analytic application. 
     
     
         16 . The method of  claim 9 , further comprising receiving new predictions that are made by the analytic application, and determining a new accuracy value of the newly received predictions made by the analytic application based on the selected performance metric, the newly received predictions made by the analytic application, and the expected predictive output of the analytic application. 
     
     
         17 . A non-transitory computer-readable medium storing instructions which when executed by a processor cause a computer to perform a method comprising:
 receiving predictions made by an analytic application;   selecting at least one performance metric for evaluating an accuracy of the analytic application based on a type of the analytic application;   determining an accuracy value of the predictions made by the analytic application based on the selected performance metric, the predictions made by the analytic application, and an expected predictive output of the analytic application;   determining that a predictive capability of the analytic application has deteriorated based on the determined accuracy value; and   transmitting an alert to a user device associated with the analytic application.   
     
     
         18 . The non-transitory computer-readable medium of  claim 17 , wherein the determining comprises determining how accurate the predictions made by the analytic application are via execution of the selected at least one performance metric. 
     
     
         19 . The non-transitory computer-readable medium of  claim 17 , wherein the accuracy value of the analytic application identifies how accurate the analytic application is for a predetermined period of time. 
     
     
         20 . The non-transitory computer-readable medium of  claim 17 , wherein the method further comprises displaying, via a user interface, a list of performance metrics associated with the analytic application that are capable of being selected by a user to measure the predictive accuracy of the analytic application.