Visual graphical user interface verification

Abstract

An automated testing system is described for efficient visual verification of graphical user interfaces of software applications. A pattern is formed for the user interface of a page of the application indicating regions of the page where user interface elements should be located and identifying which user interface element should be located in which region. During test execution, image recognition is performed using previously stored snapshots of user interface elements to determine whether the application's user interface elements appear in correct positions on the page.

What is claimed is: 
     
       1. A method for automated testing of a user interface (UI) of a software application, the method comprising stages executed on a processor of a computing device, the stages comprising:
 executing a test on a page of the UI of the application, the test configured to determine whether a plurality of UI elements appear in correct respective visual locations, relative to one another, according to a stored, previously constructed pattern corresponding to the page; 
 retrieving the stored, previously constructed pattern from a database, wherein the pattern comprises a plurality of snapshots, each snapshot including at least one of the plurality of UI elements, and wherein the pattern also comprises location information corresponding to a respective visual location of each of the plurality of UI elements on the page relative to one another; and 
 verifying that the page of the UI of the application matches the stored, previously constructed pattern by identifying at least two of the plurality of UI elements based on the respective snapshots of the at least two UI elements and determining whether the at least two UI elements are in the same respective visual locations on the page, relative to one another, as in the pattern, based on the location information corresponding to the at least two UI elements. 
 
     
     
       2. The method of  claim 1 , wherein the location information further comprises information defining, for each of the plurality of UI elements, a region corresponding to the UI element; and
 wherein verifying that the page of the UI of the application matches the pattern further comprises performing image recognition of a snapshot of each of the plurality of UI elements to determine if the UI element is located within the region of the UI corresponding to the UI element. 
 
     
     
       3. The method of  claim 2 , wherein performing image recognition of the snapshot of each of the plurality of UI elements further comprises checking whether a matching image to the snapshot is located within the region of the UI corresponding to the UI element. 
     
     
       4. The method of  claim 2 , wherein the information defining, for each of the plurality of UI elements, a region corresponding to the UI element further comprises:
 information defining a size and shape of the region; and 
 information defining the location of the region on the page. 
 
     
     
       5. The method of  claim 2 , further comprising:
 identifying the location of a base UI element on the page, wherein the base UI element is one of the plurality of UI elements, by performing image recognition of a snapshot of the base UI element on the page, wherein the location information defining the region corresponding to at least one other UI element in the plurality of UI elements comprises information defining the location of the at least one other UI element with respect to the location of the base UI element. 
 
     
     
       6. The method of  claim 2 , wherein the location information comprises the relative position of at least two regions with respect to each other. 
     
     
       7. The method of  claim 1 , wherein each of the plurality of UI elements is a non-text UI element. 
     
     
       8. A computing device, comprising:
 at least one processor; and 
 memory including instructions that, when executed by the at least one processor, cause the computing device to:
 execute a test on a page of a user interface (UI) of an application, the test configured to determine whether a plurality of UI elements appear in correct respective visual locations, relative to one another, according to a stored, previously constructed pattern corresponding to the page; 
 retrieve the stored, previously constructed pattern from a database, wherein the pattern comprises a plurality of snapshots, each snapshot including at least one of the plurality of UI elements, and wherein the pattern also comprises location information corresponding to a respective visual location of each of the plurality of UI elements on the page relative to one another; and 
 verify that the page of the UI of the application matches the stored, previously constructed pattern by identifying at least two of the plurality of UI elements based on the respective snapshots of the at least two UI elements and determining whether the at least two UI elements are in the same respective visual locations on the page, relative to one another, as in the pattern, based on the location information corresponding to the at least two UI elements. 
 
 
     
     
       9. The computing device of  claim 8 , wherein the location information further comprises information defining, for each of the plurality of UI elements, a region corresponding to the UI element; and
 verifying that the page of the UI of the application matches the pattern further comprises performing image recognition of a snapshot of each of the plurality of UI elements to determine if the UI element is located within the region of the UI corresponding to the UI element. 
 
     
     
       10. The computing device of  claim 9 , wherein performing image recognition of the snapshot of each of the plurality of UI elements further comprises checking whether a matching image to the snapshot is located within the region of the UI corresponding to the UI element. 
     
     
       11. The computing device of  claim 9 , wherein the information defining, for each of the plurality of UI elements, a region corresponding to the UI element further comprises:
 information defining a size and shape of the region; and 
 information defining the location of the region on the page. 
 
     
     
       12. The computing device of  claim 9 , wherein the memory further includes instructions that when executed by the at least one processor, cause the computing device to:
 identify the location of a base UI element on the page, wherein the base UI element is one of the plurality of UI elements, by performing image recognition of a snapshot of the base UI element on the page, wherein the location information defining the region corresponding to at least one other UI element in the plurality of UI elements comprises information defining the location of the at least one other UI element with respect to the location of the base UI element. 
 
     
     
       13. The computing device of  claim 9 , wherein the location information comprises the relative position of at least two regions with respect to each other. 
     
     
       14. The computing device of  claim 8 , wherein each of the plurality of UI elements is a non-text UI element. 
     
     
       15. A non-transitory computer readable storage medium comprising one or more sequences of instructions, the instructions when executed by one or more processors causing the one or more processors to execute the operations of:
 executing a test on a page of a user interface (UI) of an application, the test configured to determine whether a plurality of UI elements appear in correct respective visual locations, relative to one another, according to a stored, previously constructed pattern corresponding to the page; 
 retrieving the stored, previously constructed pattern from a database, wherein the pattern comprises a plurality of snapshots, each snapshot including at least one of the plurality of UI elements, and wherein the pattern also comprises location information corresponding to a respective visual location of each of the plurality of UI elements on the page relative to one another; and 
 verifying that the page of the UI of the application matches the stored, previously constructed pattern by identifying at least two of the plurality of UI elements based on the respective snapshots of the at least two UI elements and determining whether the at least two UI elements are in the same respective visual locations on the page, relative to one another, as in the pattern, based on the location information corresponding to the at least two UI elements. 
 
     
     
       16. The non-transitory computer readable storage medium of  claim 15 , wherein the location information further comprises information defining for each of the plurality of UI elements a region corresponding to the UI element; and
 verifying that the page of the UI of the application matches the pattern further comprises performing image recognition of a snapshot of each of the plurality of UI elements to determine if the UI element is located within the region of the UI corresponding to the UI element. 
 
     
     
       17. The non-transitory computer readable storage medium of  claim 16 , wherein performing image recognition of the snapshot of each of the plurality of UI elements further comprises checking whether a matching image to the snapshot is located within the region of the UI corresponding to the UI element. 
     
     
       18. The non-transitory computer readable storage medium of  claim 16 , wherein the information defining, for each of the plurality of UI elements, a region corresponding to the UI element further comprises:
 information defining a size and shape of the region; and 
 information defining the location of the region on the page. 
 
     
     
       19. The non-transitory computer readable storage medium of  claim 16 , further comprising instructions that when executed by the one or more processors cause the one or more processors to execute the operations of:
 identifying the location of a base UI element on the page, wherein the base UI element is one of the plurality of UI elements, by performing image recognition of a snapshot of the base UI element on the page, wherein the location information defining the region corresponding to at least one other UI element in the plurality of UI elements further comprises information defining the location of the at least one other UI element with respect to the location of the base UI element. 
 
     
     
       20. The non-transitory computer readable storage medium of  claim 16 , wherein the location information comprises the relative position of at least two regions with respect to each other.