Video-based user interface aplication testing

Abstract

Described herein are methods, computer program products, and computer systems for video-based user interface (UI) application testing. The method includes receiving first test video data corresponding to test video images of an application executing on a first UI, generating the test video images on a first display, generating application video images on a second display. Further, the method may include determining that a first frame of the test video images and a second frame of the application video images fail to satisfy a predetermined similarity threshold, generating a third UI comprising the second frame of the second UI on a third display, receiving user inputs at the third UI for a first duration, capturing replacement test video images from the third display for the first duration, and generating second test video images, wherein the first frame of the test video images is replaced by the replacement test video images.

What is claimed is: 
     
         1 . A computer-implemented method, comprising:
 determining, by one or more processors, that a first portion of test video images and a second portion of application video images are different;   generating, by one or more processors, a UI comprising the second portion on the user device based on the application video data;   receiving, by one or more processors, user inputs for a first duration;   capturing, by one or more processors, replacement test video images based on the user inputs for the first duration; and   generating, by one or more processors, second test video images based on the test video images and the replacement test video images, wherein the first portion of the test video images is replaced by the replacement test video images.   
     
     
         2 . The computer-implemented method of  claim 1 , further comprising:
 determining, by one or more processors, a first score for the first portion and a second score for the second portion;   generating, by one or more processors, a first similarity threshold based on a comparison between the first score and the second score; and   determining, by one or more processors, that the first portion and the second portion is different if the first similarity threshold is less than a predetermined threshold.   
     
     
         3 . The computer-implemented method of  claim 1 , wherein the user inputs correspond to specific instructions executed by a user to click one or more user-selectable icons. 
     
     
         4 . The computer-implemented method of  claim 1 , wherein the user inputs correspond to specific instructions executed by a user to enter text characters into one or more fields. 
     
     
         5 . The computer-implemented method of  claim 1 , wherein the user inputs correspond to specific instructions executed by a user to visually verify one or more UI icons. 
     
     
         6 . The computer-implemented method of  claim 1 , wherein the first duration begins at a first time after the UI is generated and ends at a second time after a period of times has elapsed without user input detection. 
     
     
         7 . The computer-implemented method of  claim 6 , wherein the first time is less than two seconds and the second time is less than two seconds. 
     
     
         8 . A computer-program product, the computer program product comprising:
 one or more computer readable storage media and program instructions collectively stored on the one or more computer readable storage media, the stored program instructions comprising program instructions to perform a computer-implemented method comprising:   program instructions to determine that a first portion of test video images and a second portion of application video images are different;   program instructions to generate a UI comprising the second portion on the user device based on the application video data;   program instructions to receive user inputs for a first duration;   program instructions to capture replacement test video images based on the user inputs for the first duration; and   program instructions to generate second test video images based on the test video images and the replacement test video images, wherein the first portion of the test video images is replaced by the replacement test video images.   
     
     
         9 . The computer program product of  claim 8 , further comprising:
 program instructions, collectively stored on the one or more computer readable storage media, to determine a first score for the first portion and a second score for the second portion;   program instructions, collectively stored on the one or more computer readable storage media, to generate a first similarity threshold based on a comparison between the first score and the second score; and   program instructions, collectively stored on the one or more computer readable storage media, to determine that the first portion and the second portion is different if the first similarity threshold is less than a predetermined threshold.   
     
     
         10 . The computer program product of  claim 8 , wherein the user inputs correspond to specific instructions executed by a user to click one or more user-selectable icons. 
     
     
         11 . The computer program product of  claim 8 , wherein the user inputs correspond to specific instructions executed by a user to enter text characters into one or more fields. 
     
     
         12 . The computer program product of  claim 8 , wherein the user inputs correspond to specific instructions executed by a user to visually verify one or more UI icons. 
     
     
         13 . The computer program product of  claim 8 , wherein the first duration begins at a first time after the UI is generated and ends at a second time after a period of times has elapsed without user input detection. 
     
     
         14 . The computer program product of  claim 13 , wherein the first time is less than two seconds and the second time is less than two seconds. 
     
     
         15 . A computer system, the computer system comprising:
 one or more computer processors, one or more computer readable storage media, and program instructions collectively stored on the one or more computer readable storage media for execution by at least one of the one or more computer processors, the program instructions comprising:   program instructions to determine that a first portion of test video images and a second portion of application video images are different;   program instructions to generate a UI comprising the second portion on the user device based on the application video data;   program instructions to receive user inputs for a first duration;   program instructions to capture replacement test video images based on the user inputs for the first duration; and   program instructions to generate second test video images based on the test video images and the replacement test video images, wherein the first portion of the test video images is replaced by the replacement test video images.   
     
     
         16 . The computer system of  claim 15 , further comprising:
 program instructions, collectively stored on the one or more computer readable storage media for execution by at least one of the one or more computer processors, to determine a first score for the first portion and a second score for the second portion;   program instructions, collectively stored on the one or more computer readable storage media for execution by at least one of the one or more computer processors, to generate a first similarity threshold based on a comparison between the first score and the second score; and   program instructions, collectively stored on the one or more computer readable storage media for execution by at least one of the one or more computer processors, to determine that the first portion and the second portion is different if the first similarity threshold is less than a predetermined threshold.   
     
     
         17 . The computer system of  claim 15 , wherein the user inputs correspond to specific instructions executed by a user to click one or more user-selectable icons. 
     
     
         18 . The computer system of  claim 15 , wherein the user inputs correspond to specific instructions executed by a user to enter text characters into one or more fields. 
     
     
         19 . The computer system of  claim 15 , wherein the user inputs correspond to specific instructions executed by a user to visually verify one or more UI icons. 
     
     
         20 . The computer system of  claim 15 , wherein the first duration begins at a first time after the UI is generated and ends at a second time after a period of times has elapsed without user input detection.