Graphical element detection using a combined series and delayed parallel execution unified target technique, a default graphical element detection technique, or both

Abstract

Graphical element detection using a combined series and delayed parallel execution unified target technique that potentially uses a plurality of graphical element detection techniques, performs default user interface (UI) element detection technique configuration at the application and/or UI type level, or both, is disclosed. The unified target merges multiple techniques of identifying and automating UI elements into a single cohesive approach. A unified target descriptor chains together multiple types of UI descriptors in series, uses them in parallel, or uses at least one technique first for a period of time and then runs at least one other technique in parallel or alternatively if the first technique does not find a match within the time period.

1 . A non-transitory computer-readable medium storing a computer program, the computer program configured to cause at least one processor to:
 analyze a user interface (UI) at runtime to identify UI element attributes;   compare the UI element attributes to UI descriptor attributes for an activity of a robotic process automation (RPA) workflow using one or more initial graphical element detection techniques; and   when a match is not found using the one or more initial graphical element detection techniques during a first time period:
 execute one or more additional graphical element detection techniques instead of the one or more initial graphical element detection techniques. 
   
     
     
         2 . The non-transitory computer-readable medium of  claim 1 , wherein when a match is not found using the one or more additional graphical element detection techniques during a second time period, the computer program is further configured to cause the at least one processor to:
 execute one or more supplemental graphical element detection techniques instead of the one or more initial graphical element detection techniques and the one or more additional graphical element detection techniques.   
     
     
         3 . The non-transitory computer-readable medium of  claim 1 , wherein when a match is found, the computer program is further configured to cause the at least one processor to:
 take an action associated with the activity involving the UI element.   
     
     
         4 . The non-transitory computer-readable medium of  claim 1 , wherein the process of  claim 1  is repeated for at least one additional activity. 
     
     
         5 . The non-transitory computer-readable medium of  claim 1 , wherein the UI descriptor attributes comprise two or more of selector attributes, computer vision (CV) attributes, image matching attributes, and optical character recognition (OCR) attributes. 
     
     
         6 . The non-transitory computer-readable medium of  claim 1 , wherein the one or more initial graphical element detection techniques, the one or more additional graphical element detection techniques, or both, comprise two or more of a selector technique, a computer vision (CV) technique, an image matching technique, and an optical character (OCR) technique. 
     
     
         7 . The non-transitory computer-readable medium of  claim 1 , wherein the computer program is or comprises an RPA robot. 
     
     
         8 . A computer-implemented method, comprising:
 analyzing a user interface (UI) at runtime to identify UI element attributes, by a robotic process automation (RPA) robot executing on a computing system;   comparing the UI element attributes to UI descriptor attributes for an activity of an RPA workflow using one or more initial graphical element detection techniques, by the RPA robot; and   when a match is not found using the one or more initial graphical element detection techniques during a first time period:
 executing one or more additional graphical element detection techniques instead of the one or more initial graphical element detection techniques, by the RPA robot. 
   
     
     
         9 . The computer-implemented method of  claim 8 , wherein when a match is not found using the one or more additional graphical element detection techniques during a second time period, the method further comprises:
 executing, by the RPA robot, one or more supplemental graphical element detection techniques instead of the one or more initial graphical element detection techniques and the one or more additional graphical element detection techniques.   
     
     
         10 . The computer-implemented method of  claim 8 , wherein when a match is found, the method further comprises:
 taking an action associated with the activity involving the UI element, by the RPA robot.   
     
     
         11 . The computer-implemented method of  claim 8 , wherein the process of  claim 8  is repeated for at least one additional activity. 
     
     
         12 . The computer-implemented method of  claim 8 , wherein the UI descriptor attributes comprise two or more of selector attributes, computer vision (CV) attributes, image matching attributes, and optical character recognition (OCR) attributes. 
     
     
         13 . The computer-implemented method of  claim 8 , wherein the one or more initial graphical element detection techniques, the one or more additional graphical element detection techniques, or both, comprise two or more of a selector technique, a computer vision (CV) technique, an image matching technique, and an optical character (OCR) technique. 
     
     
         14 . A computing system, comprising:
 memory storing computer program instructions; and   at least one processor configured to execute the computer program instructions, wherein the computer program instructions are configured to cause the at least one processor to:
 analyze a user interface (UI) at runtime to identify UI element attributes, 
 compare the UI element attributes to UI descriptor attributes for an activity of a robotic process automation (RPA) workflow using one or more initial graphical element detection techniques, and 
 when a match is not found using the one or more initial graphical element detection techniques during a first time period:
 execute one or more additional graphical element detection techniques instead of the one or more initial graphical element detection techniques. 
 
   
     
     
         15 . The computing system of  claim 14 , wherein when a match is not found using the one or more additional graphical element detection techniques during a second time period, the computer program instructions are further configured to cause the at least one processor to:
 execute one or more supplemental graphical element detection techniques instead of the one or more initial graphical element detection techniques and the one or more additional graphical element detection techniques.   
     
     
         16 . The computing system of  claim 14 , wherein when a match is found, the computer program instructions are further configured to cause the at least one processor to:
 take an action associated with the activity involving the UI element.   
     
     
         17 . The computing system of  claim 14 , wherein the process of  claim 14  is repeated for at least one additional activity. 
     
     
         18 . The computing system of  claim 14 , wherein the UI descriptor attributes comprise two or more of selector attributes, computer vision (CV) attributes, image matching attributes, and optical character recognition (OCR) attributes. 
     
     
         19 . The computing system of  claim 14 , wherein the one or more initial graphical element detection techniques, the one or more additional graphical element detection techniques, or both, comprise two or more of a selector technique, a computer vision (CV) technique, an image matching technique, and an optical character (OCR) technique. 
     
     
         20 . The computing system of  claim 14 , wherein the computer program instructions are or comprise an RPA robot.