Method and system for estimating the accuracy of inference algorithms using the self-consistency methodology

Abstract

The present invention provides a method for measuring the self-consistency of inference algorithms. The present invention provides a method for measuring the accuracy of an inference algorithm that does not require comparison to ground truth. Rather, the present invention pertains to a method for measuring the accuracy of an inference algorithm by comparing the outputs of the inference algorithm against each other. Essentially, the present invention looks at how well the algorithm applied to many of the different observations gives the same answer. In particular, the present invention provides a method that is not time and labor intensive and is cost effective.

1. A method for estimating an accuracy of an inference process, said method comprising the steps of:
 a) collecting a plurality of observations of a scene, wherein said scene is within a class of scenes and observational conditions; 
 b) applying said inference process to each observation of said scene; 
 c) measuring a self-consistency of said inference process by comparing a plurality of hypotheses of said inference process as a function of a nature of said observation; and 
 d) for a plurality of hypotheses of said scene, incrementing a histogram of a function of an estimate of an attribute of a hypothesis of said scene, said hypothesis normalized by the function of said estimate, resulting in a statistical analysis of the self-consistency of said inference process. 
 
     
     
       2. The method as recited in  claim 1  wherein said scene is a static scene. 
     
     
       3. The method as recited in  claim 1  wherein step d) comprises the step of
 d1) conditionalizing said histogram on a score, said score being an appraisal of the confidence of the accuracy of said estimate. 
 
     
     
       4. The method as recited in  claim 3  further comprising the step of:
 e) adjusting the internal parameters of said inference process based on said statistical analysis. 
 
     
     
       5. The method as recited in  claim 3  further comprising the step of:
 e) comparing different inference processes based on said statistical analysis. 
 
     
     
       6. The method as recited in  claim 1  further comprising the step of:
 e) adjusting the internal parameters of said inference process based on said statistical analysis. 
 
     
     
       7. The method as recited in  claim 1  further comprising the step of:
 e) comparing different inference processes based on said statistical analysis. 
 
     
     
       8. A computer-readable medium having stored thereon instructions for causing a computer to implement a process for estimating an accuracy of an inference process to perform the steps of:
 a) collecting a plurality of observations of a scene, wherein said scene is within a class of scenes and observational conditions; 
 b) applying said inference process to each observation of said scene; 
 c) measuring a self-consistency of said inference process by comparing a plurality of hypotheses of said inference process as a function of a nature of said observation; and 
 d) for a plurality of hypotheses of said scene, incrementing a histogram of a function of an estimate of an attribute of a hypothesis of said scene, said hypothesis normalized by the function of said estimate, resulting in a statistical analysis of the self-consistency of said inference process. 
 
     
     
       9. The computer-readable medium of  claim 8  wherein said scene is a static scene. 
     
     
       10. The computer-readable medium of  claim 8  wherein said instructions therein causes a computer to perform the step of:
 d1) conditionalizing said histogram on a score, said score being an appraisal of the confidence of the accuracy of said estimate. 
 
     
     
       11. The computer-readable medium of  claim 10  wherein said instructions therein causes a computer to perform the step of:
 e) adjusting the internal parameters of said inference process based on said statistical analysis. 
 
     
     
       12. The computer-readable medium of  claim 10  wherein said instructions therein causes a computer to perform the step of:
 e) comparing different inference processes based on said statistical analysis. 
 
     
     
       13. The computer-readable medium of  claim 8  wherein said instructions therein causes a computer to perform the step of:
 e) adjusting the internal parameters of said inference process based on said statistical analysis. 
 
     
     
       14. The computer-readable medium of  claim 8  wherein said instructions therein causes a computer to perform the step of:
 e) comparing different inference processes based on said statistical analysis. 
 
     
     
       15. An computer system comprising:
 a bus; 
 a processor coupled to said bus; and 
 a computer-readable memory unit coupled to said bus; 
 said processor for performing a method for estimating an accuracy of an inference process, said method comprising the steps of:
 a) collecting a plurality of observations of a scene, wherein said scene is within a class of scenes and observational conditions; 
 b) applying said inference process to each observation of said scene; 
 c) measuring a self-consistency of said inference process by comparing a plurality of hypotheses of said inference process as a function of a nature of said observation; and 
 d) for a plurality of hypotheses of said scene, incrementing a histogram of a function of an estimate of an attribute of a hypothesis of said scene, said hypothesis normalized by the function of said estimate, resulting in a statistical analysis of the self-consistency of said inference process. 
 
 
     
     
       16. The computer system of  claim 15 , wherein said scene is a static scene. 
     
     
       17. The computer system of  claim 15  wherein said processor performs said method for estimating an accuracy of an inference process, further comprising the step of:
 d1) conditionalizing said histogram on a score, said score being an appraisal of the confidence of the accuracy of said estimate. 
 
     
     
       18. The computer system of  claim 17  wherein said processor performs said method for estimating an accuracy of an inference process, further comprising the step of:
 e) adjusting the internal parameters of said inference process based on said statistical analysis. 
 
     
     
       19. The computer system of  claim 17  wherein said processor performs said method for estimating an accuracy of an inference process, further comprising the step of:
 e) comparing different inference processes based on said statistical analysis. 
 
     
     
       20. The computer system of  claim 15  wherein said processor performs said method for estimating an accuracy of an inference process, further comprising the step of:
 e) adjusting the internal parameters of said inference process based on said statistical analysis. 
 
     
     
       21. The computer system of  claim 15  wherein said processor performs said method for estimating an accuracy of an inference process, further comprising the step of:
 e) comparing different inference processes based on said statistical analysis. 
 
     
     
       22. A computing device for estimating an accuracy of an inference process, wherein the computing device comprises a processor and a memory, wherein the computing device is configured to:
 collect a plurality of observations of a scene;   apply an inference process to each observation of the scene;   measure a self-consistency of the inference process; and   obtain a statistical analysis of the self-consistency of the inference process.   
     
     
       23. The computing device of claim 22, wherein the scene is within a class of scenes and observational conditions. 
     
     
       24. The computing device of claim 22, wherein measuring the self-consistency of the inference process comprises:
 comparing a plurality of hypotheses of the inference process as a function of a nature of the observation.   
     
     
       25. The computing device of claim 22, wherein obtaining a statistical analysis of the self-consistency of the inference process comprises:
 for a plurality of hypothesis of the scene, incrementing a histogram of a function of an estimate of an attribute of a histogram of the scene, the hypothesis normalized by the function of the estimate.   
     
     
       26. The computing device of claim 25, further configured to:
 conditionalize the histogram on a score, wherein the score comprises an appraisal of the confidence of the accuracy of the estimate.   
     
     
       27. The computing device of claim 25, further configured to:
 adjust the internal parameters of the inference process based on the statistical analysis.   
     
     
       28. The computing device of claim 25, further configured to:
 compare different inference processes based on the statistical analysis.   
     
     
       29. The computing device of claim 22, further configured to:
 adjust the internal parameters of the inference process based on the statistical analysis.   
     
     
       30. The computing device of claim 22, further configured to:
 compare different inference processes based on the statistical analysis.   
     
     
       31. An apparatus, comprising:
 means for collecting a plurality of observations of a scene;   means for applying an inference process to each observation of the scene;   means for measuring a self-consistency of the inference process; and   means for obtaining a statistical analysis of the self-consistency of the inference process.   
     
     
       32. The apparatus of claim 31, wherein the scene is within a class of scenes and observational conditions. 
     
     
       33. The apparatus of claim 31, wherein the means for measuring the self-consistency of the inference process comprises:
 means for comparing a plurality of hypotheses of the inference process as a function of a nature of the observation.   
     
     
       34. The apparatus of claim 31, wherein means for obtaining a statistical analysis of the self-consistency of the inference process comprises:
 for a plurality of hypotheses of the scene, means for incrementing a histogram of a function of an estimate of an attribute of a hypothesis of the scene, the hypothesis normalized by the function of the estimate.   
     
     
       35. The apparatus of claim 34, further comprising:
 means for conditionalizing the histogram on a score, wherein the score comprises an appraisal of the confidence of the accuracy of the estimate.   
     
     
       36. The apparatus of claim 34, further comprising:
 means for adjusting the internal parameters of the inference process based on the statistical analysis.   
     
     
       37. The apparatus of claim 34, further comprising:
 means for comparing different inference processes based on the statistical analysis.   
     
     
       38. The apparatus of claim 31, further comprising:
 means for adjusting the internal parameters of the inference process based on the statistical analysis.   
     
     
       39. The apparatus of claim 31, further comprising:
 means for comparing different inference processes based on the statistical analysis.