Systems and methods for emulating the behavior of a user in a computer-human interaction environment

Abstract

A computer-implemented method for emulating the behavior of a user in a computer-human interaction environment is described. An image of a window and data relating to positions of clicks executed within the image are received. A probabilistic model is created to estimate a probability of a click being executed in a region of a window. Clicks, in accordance with the probabilistic model, are executed within windows associated with a plurality of applications. A clicks distribution model is created based on the position of the clicks executed within the windows of the plurality of applications. Clicks, in accordance with the clicks distribution model, are executed within a window associated with an application being tested.

1. A computer-implemented method for emulating the behavior of a user in a computer-human interaction environment, comprising:
 receiving an image of a window and data relating to positions of clicks executed within the image; 
 extracting one or more features from the image related to human visual perception; 
 combining the one or more extracted features with the data relating to the positions of the clicks executed within the image; 
 creating a probabilistic model from the combination to estimate a probability of a click being executed in a region of a window; 
 executing clicks, in accordance with the probabilistic model, within windows associated with a plurality of applications; 
 creating a probability distribution model based on the position of the clicks executed within the windows of the plurality of applications; and 
 executing clicks, in accordance with the probability distribution model, within a window associated with an application being tested. 
 
     
     
       2. The method of  claim 1 , wherein the one or more windows comprise a user interface. 
     
     
       3. The method of  claim 1 , further comprising applying a reinforcement learning algorithm to process data collected from the application and the probabilistic model to produce a second probabilistic model, wherein the second model incorporates initial information regarding images, sequences of clicks, and the behavior of the application. 
     
     
       4. The method of  claim 3 , wherein the sequence of clicks maintain the execution of the application for a specified length of time. 
     
     
       5. The method of  claim 3 , wherein the sequence of clicks cause a predetermined activity from the application. 
     
     
       6. The method of  claim 3 , wherein the sequence of clicks executed within the window of the application create a specified file. 
     
     
       7. The method of  claim 1 , wherein the application being tested is a potentially malicious application. 
     
     
       8. The method of  claim 1 , wherein the extracted features of the image comprise contrast zones, textures, contours, or shapes within the image. 
     
     
       9. A computer system configured to emulate the behavior of a user in a computer-human interaction environment, comprising:
 a processor; 
 memory in electronic communication with the processor 
 instructions stored in the memory, the instructions being executable by the processor to:
 receive an image of a window and data relating to positions of clicks executed within the image; 
 extract one or more features from the image related to human visual perception; 
 combine the one or more extracted features with the data relating to the positions of the clicks executed within the image; 
 create a probabilistic model from the combination to estimate a probability of a click being executed in a region of a window; 
 execute clicks, in accordance with the probabilistic model, within windows associated with a plurality of applications; 
 create a probability distribution model based on the position of the clicks executed within the windows of the plurality of applications; and 
 execute clicks, in accordance with the probability distribution model, within a window associated with an application being tested. 
 
 
     
     
       10. The computer system of  claim 9 , wherein the one or more windows comprise a user interface. 
     
     
       11. The computer system of  claim 9 , wherein the processor is further configured to apply a reinforcement learning algorithm to process data collected from the application and the probabilistic model to produce a second probabilistic model, wherein the second model incorporates initial information regarding images, sequences of clicks, and the behavior of the application. 
     
     
       12. The computer system of  claim 11 , wherein the sequence of clicks maintain the execution of the application for a specified length of time. 
     
     
       13. The computer system of  claim 11 , wherein the sequence of clicks cause a predetermined activity from the application. 
     
     
       14. The computer system of  claim 11 , wherein the sequence of clicks within the window of the application create a specified file. 
     
     
       15. The computer system of  claim 9 , wherein the application being tested is a potentially malicious application. 
     
     
       16. A computer-program product for emulating the behavior of a user in a computer-human interaction environment, the computer-program product comprising a non-transitory computer-readable medium having instructions thereon, the instructions being executable by a processor to:
 receive an image of a window and data relating to positions of clicks executed within the image; 
 extract one or more features from the image related to human visual perception; 
 combine the one or more extracted features with the data relating to the positions of the clicks executed within the image; 
 create a probabilistic model from the combination to estimate a probability of a click being executed in a region of a window; 
 execute clicks, in accordance with the probabilistic model, within windows associated with a plurality of applications; 
 create a probability distribution model based on the position of the clicks executed within the windows of the plurality of applications; and 
 execute clicks, in accordance with the probability distribution model, within a window associated with an application being tested.