Managing automated and manual application testing

Abstract

An application for which approval is requested is identified and multiple automated tests are applied to the application in groups of automated tests. Each of the groups of automated tests includes multiple ones of the multiple automated tests. If one or more automated tests in a group of automated tests returns an inconclusive result, then a manual check is initiated for the application based on the one or more automated tests that returned the inconclusive result. If one or more automated tests in a group, or a manual test applied in the manual check, returns a fail result then an indication that the application is rejected is returned, the indication that the application is rejected including an identification of why the application is rejected. If none of the multiple automated tests returns a fail result, then a manual testing phase is initiated.

What is claimed is: 
     
       1. A method comprising:
 identifying, via a testing system that includes a processor, an application for which approval is requested; 
 testing the application by applying multiple automated tests to the application in groups of automated tests, each of the groups of automated tests including multiple ones of the multiple automated tests; 
 identifying, prior to completing all of the multiple automated tests and for one or more of the multiple automated tests that returns an inconclusive result, that a manual check is to be performed for the application based on the one or more automated tests, the inconclusive result indicating that the one or more automated tests cannot determine whether the application is to receive a fail result or a pass result; 
 for each of the groups of automated tests causing, after applying the automated tests in the group, an indication that the application is rejected to be returned if one or more of the automated tests in the group returns a fail result, the indication that the application is rejected including an identification of why the application is rejected; 
 identifying, if none of the multiple automated tests returns a fail result, that a manual testing phase is to be performed; and 
 causing, after the manual testing phase, an indication that the application is approved to be returned if none of multiple manual tests in the manual testing phase returns a fail result. 
 
     
     
       2. A method as recited in  claim 1 , further comprising causing the indication that the application is rejected to be returned when two or more of the automated tests return a fail result, the indication that the application is rejected including an identification of each of the two or more of the automated tests that returned a fail result or an identification, returned by each of the two or more automated tests that returned a fail result, of why the automated test returned a fail result. 
     
     
       3. A method as recited in  claim 1 , wherein the identification of why the application is rejected includes an identification of each of the one or more automated tests in the group that returned a fail result. 
     
     
       4. A method as recited in  claim 1 , wherein the identification of why the application is rejected includes an identification, returned by each of the one or more automated tests in the group that returned a fail result, of why the automated test returned a fail result. 
     
     
       5. A method as recited in  claim 1 , further comprising, if one or more of the multiple automated tests returns an inconclusive result:
 wherein the manual check includes one or more manual tests; 
 continuing applying the multiple automated tests to the application in groups of automated tests if the one or more manual tests return a pass result; and 
 causing an indication that the application is rejected to be returned if at least one of the one or more manual tests returns a fail result. 
 
     
     
       6. A method as recited in  claim 1 , further comprising determining, based on the one or more automated tests that returned the inconclusive result, one or more manual tests to be applied as part of the manual check. 
     
     
       7. A method as recited in  claim 1 , wherein the multiple automated tests are grouped into the groups of automated tests based on types of each of the multiple automated tests. 
     
     
       8. A method as recited in  claim 1 , wherein applying the multiple automated tests to the application in groups of automated tests comprises applying tests related to security checking of the application prior to applying other types of automated tests. 
     
     
       9. A method as recited in  claim 1 , further comprising routing, as part of the manual testing phase, the application to one or more entities to perform one or more manual tests, wherein the one or more entities are determined based at least in part on characteristics of the application. 
     
     
       10. A method as recited in  claim 1 , wherein the approval comprises approval to be included in an online store. 
     
     
       11. A method as recited in  claim 1 , wherein identifying that the manual testing phase is to be performed is responsive to identifying that no manual tests of the manual check return a fail result. 
     
     
       12. One or more computer readable storage media having stored thereon multiple instructions that, when executed by one or more processors, cause the one or more processors to:
 identify an application for which approval is requested; 
 apply multiple automated tests to the application; 
 determine, prior to completing all of the multiple automated tests and for one or more of the multiple automated tests that returns an inconclusive result, that a manual check is to be performed for the application based on the one or more automated tests, the inconclusive result indicating that the one or more automated tests cannot determine whether the application is to receive a fail result or a pass result; 
 cause an indication that the application is rejected to be returned if one or more of the multiple automated tests returns a fail result; and 
 cause, after a manual testing phase, an indication that the application is approved to be returned if none of multiple manual tests in the manual testing phase returns a fail result. 
 
     
     
       13. One or more computer readable storage media as recited in  claim 12 , wherein the multiple instructions further cause the one or more processors to cause an indication that the application is rejected to be returned if one or more manual tests of the manual check returns a fail result. 
     
     
       14. One or more computer readable storage media as recited in  claim 12 , wherein the multiple instructions further cause the one or more processors to continue to apply the multiple automated tests to the application if none of the manual tests of the manual check returns a fail result. 
     
     
       15. One or more computer readable storage media as recited in  claim 12 , wherein to apply the multiple automated tests to the application is to apply the multiple automated tests in groups of automated tests, each of the groups of automated tests including two or more of the multiple automated tests. 
     
     
       16. One or more computer readable storage media as recited in  claim 12 , wherein to determine that the manual check is to be performed is based on two or more automated tests that each return an inconclusive result, the manual check performed by applying multiple manual tests including at least one manual test determined based at least in part on each of the two or more automated tests, and wherein the multiple instructions further cause the one or more processors to:
 cause an indication that the application is rejected to be returned if one or more of the multiple manual tests returns a fail result; and 
 continue to apply the multiple automated tests to the application if none of the multiple manual tests returns a fail result. 
 
     
     
       17. One or more computer readable storage media as recited in  claim 12 , wherein the multiple instructions further cause the one or more processors to determine, if none of the multiple automated tests returns a fail result and no manual tests of the manual check returns a fail result, that the manual testing phase is to be performed. 
     
     
       18. One or more computer readable storage media as recited in  claim 17 , wherein the multiple instructions further cause the one or more processors to route, as part of the manual testing phase, the application to one or more entities to perform one or more manual tests, wherein the one or more entities are determined based at least in part on characteristics of the application. 
     
     
       19. One or more computer readable storage media as recited in  claim 12 , wherein the multiple instructions further cause the one or more processors to determine, based at least in part on the one or more of the multiple automated tests that returns an inconclusive result, one or more manual tests to be applied as the manual check. 
     
     
       20. A method comprising:
 identifying, via a testing system that includes a processor, an application for which approval is requested; 
 testing the application by applying multiple automated tests to the application in groups of automated tests, each of the groups of automated tests including multiple ones of the multiple automated tests, and determining, prior to completing all of the multiple automated tests and for one or more of the multiple automated tests that returns an inconclusive result, that a manual check is to be performed for the application based on the one or more automated tests, the inconclusive result indicating that the one or more automated tests cannot determine whether the application is to receive a fail result or a pass result; 
 for each of the groups of automated tests causing, after applying the automated tests in the group, an indication that the application is rejected to be returned if one or more of the automated tests in the group returns a fail result or one or more of multiple tests in the manual check returns a fail result, the indication that the application is rejected including an identification of why the application is rejected; 
 determining, if none of the multiple automated tests returns a fail result, that a manual testing phase is to be performed; 
 causing, after receiving an indication that the manual testing phase is complete, an indication that the application is rejected to be returned if one or more manual tests of the manual testing phase returns a fail result, the indication that the application is rejected including an identification of why the application is rejected; and 
 causing, after receiving an indication that the manual testing phase is complete, an indication that the application is approved to be returned if none of multiple manual tests in the manual testing phase returns a fail result.