Anomaly detection and self-healing for robotic process automation via artificial intelligence / machine learning

Abstract

Anomaly detection and self-healing for robotic process automation (RPA) via artificial intelligence (AI)/machine learning (ML) is disclosed. RPA robots that utilize AI/ML models and computer vision (CV) may interpret and/or interact with most encountered graphical elements via normal learned interactions. However, such RPA robots may occasionally encounter new, unhandled anomalies where graphical elements cannot be identified and/or normal interactions will not work. Such anomalies may be processed by an anomaly handler. The RPA robots may have self-healing functionality that seeks to automatically find information that addresses anomalies.

The invention claimed is: 
     
       1. A computer-implemented method, comprising:
 executing a robotic process automation (RPA) workflow that performs a user interface (UI) automation using an artificial intelligence (AI)/machine learning (ML) model, by an RPA robot; 
 using the AI/ML model, searching for a target graphical element in the UI to be interacted with by an activity of the RPA workflow, by the RPA robot; and 
 when the target graphical element is not uniquely found by the AI/ML model or cannot be interacted with, constituting an anomaly for operation of the RPA robot, automatically attempting to correct the anomaly, by the RPA robot or the AI/ML model, wherein 
 the automatic attempt to correct the anomaly comprises determining whether one or more features differentiate the target graphical element from other similar graphical elements, analyzing graphical elements surrounding the target graphical element within a radius, utilizing an order of the graphical elements in the UI, determining whether the target graphical element has one or more different visual characteristics, changing visual characteristics of the UI and searching for the target graphical element in the UI using the changed visual characteristics, or any combination thereof. 
 
     
     
       2. The computer-implemented method of  claim 1 , wherein the automatic attempt to correct the anomaly comprises changing visual characteristics of the UI and searching for the target graphical element in the UI using the changed visual characteristics. 
     
     
       3. The computer-implemented method of  claim 1 , the automatic attempt to correct the anomaly comprises taking a default action. 
     
     
       4. The computer-implemented method of  claim 3 , wherein the default action comprises searching a UI object library or a UI object repository for one or more UI descriptors that enable interaction with the target graphical element. 
     
     
       5. The computer-implemented method of  claim 1 , wherein when the automatic attempt to correct the anomaly is successful, the method further comprises:
 providing data pertaining to the automatic correction for subsequent retraining of the AI/ML model, by the RPA robot. 
 
     
     
       6. The computer-implemented method of  claim 1 , wherein when the automatic attempt to correct the anomaly is not successful, the method further comprises:
 prompting a user for a solution to identify the target graphical element, how to interact with the target graphical element, or both, by the RPA robot. 
 
     
     
       7. The computer-implemented method of  claim 6 , wherein when the guidance provided by the user is successful in enabling the RPA robot to interact with the target graphical element, the method further comprises:
 providing data pertaining to the user solution for subsequent retraining of the AI/ML model, by the RPA robot. 
 
     
     
       8. The computer-implemented method of  claim 6 , wherein when the guidance provided by the user is not successful in enabling the RPA robot to interact with the target graphical element and the target graphical element is not necessary to complete an overall task of the RPA workflow, the method further comprises:
 attempting to continue execution of the RPA workflow, by the RPA robot. 
 
     
     
       9. The computer-implemented method of  claim 1 , wherein the automatic attempt to correct the anomaly comprises:
 attempting a self-healing process to complete missing data without user input, by the RPA robot or the AI/ML model. 
 
     
     
       10. The computer-implemented method of  claim 9 , wherein the RPA robot or the AI/ML model is configured to determine whether the self-healing process was successful by monitoring whether one or more performance metrics improve responsive to the self-healing process. 
     
     
       11. The computer-implemented method of  claim 9 , wherein when the self-healing process was not successful, the method further comprises:
 attempting one or more different techniques and monitoring whether the one or more different techniques improve the one or more performance metrics. 
 
     
     
       12. The computer-implemented method of  claim 9 , wherein the self-healing process comprises:
 polling a plurality of users to provide proposed solutions to the anomaly; and 
 selecting a most optimal solution of the proposed solutions based on one or more performance metrics. 
 
     
     
       13. The computer-implemented method of  claim 9 , wherein the attempting of the self-healing process to complete the missing data without user input comprises using an exploration phase in reinforcement learning. 
     
     
       14. A non-transitory computer-readable medium storing a computer program, the computer program configured to cause at least one processor to:
 execute a robotic process automation (RPA) workflow that performs a user interface (UI) automation using an artificial intelligence (AI)/machine learning (ML) model; 
 using the AI/ML model, search for a target graphical element in the UI to be interacted with by an activity of the RPA workflow; and 
 when the target graphical element is not uniquely found by the AI/ML model or cannot be interacted with, constituting an anomaly for operation of the RPA robot, automatically attempt to correct the anomaly, wherein 
 the automatic attempt to correct the anomaly comprises determining whether one or more features differentiate the target graphical element from other similar graphical elements, analyzing graphical elements surrounding the target graphical element within a radius, utilizing an order of the graphical elements in the UI, determining whether the target graphical element has one or more different visual characteristics, changing visual characteristics of the UI and searching for the target graphical element in the UI using the changed visual characteristics, or any combination thereof. 
 
     
     
       15. The non-transitory computer-readable medium of  claim 14 , wherein when the automatic attempt to correct the anomaly is successful, the computer program is further configured to cause the at least one processor to provide data pertaining to the automatic correction for subsequent retraining of the AI/ML model, and
 when the automatic attempt to correct the anomaly is not successful, the computer program is further configured to cause the at least one processor to prompt a user for a solution to identify the target graphical element, how to interact with the target graphical element, or both. 
 
     
     
       16. The non-transitory computer-readable medium of  claim 15 , wherein when the guidance provided by the user is successful in enabling the computer program to interact with the target graphical element, the computer program is further configured to cause the at least one processor to provide data pertaining to the user solution for subsequent retraining of the AI/ML model, and
 when the guidance provided by the user is not successful in enabling the computer program to interact with the target graphical element and the target graphical element is not necessary to complete an overall task of the RPA workflow, the computer program is further configured to cause the at least one processor to attempt to continue execution of the RPA workflow. 
 
     
     
       17. The non-transitory computer-readable medium of  claim 14 , wherein the automatic attempt to correct the anomaly comprises attempting a self-healing process to complete missing data without user input. 
     
     
       18. A computing system, comprising:
 memory storing computer program instructions; and 
 at least one processor configured to execute the computer program instructions, wherein the computer program instructions are configured to cause the at least on processor to:
 execute a robotic process automation (RPA) workflow that performs a user interface (UI) automation using an artificial intelligence (AI)/machine learning (ML) model, by an RPA robot; 
 using the AI/ML model, search for a target graphical element in the UI to be interacted with by an activity of the RPA workflow, by the RPA robot; and 
 when the target graphical element is not uniquely found by the AI/ML model or cannot be interacted with, constituting an anomaly for operation of the RPA robot, automatically attempt to correct the anomaly, by the RPA robot or the AI/ML model, wherein 
 
 the automatic attempt to correct the anomaly comprises determining whether one or more features differentiate the target graphical element from other similar graphical elements, attempting a self-healing process to complete missing data without user input, or both. 
 
     
     
       19. The computing system of  claim 18 , wherein the determining of whether the one or more features differentiate the target graphical element from the other similar graphical elements comprises analyzing graphical elements surrounding the target graphical element within a radius, utilizing an order of the graphical elements in the UI, determining whether the target graphical element has one or more different visual characteristics, or a combination thereof.