System and method for performance evaluation of probability forecast

Abstract

A method and system for probability distribution forecast evaluation are disclosed. The present disclosure is directed to embodiments of a system that evaluates probability distribution forecasts by acquiring one or more of a probability distribution forecast, a probability distribution realization, and a prior knowledge of the probability distribution forecast. The system disclosed herein may then compute an accuracy score and an information score based on the acquired forecast, realization, and prior knowledge. In evaluating the forecast, a performance score may also be computed based on the accuracy score and the information score.

1 . A probability distribution forecast evaluation system comprising:
 at least one processor;   at least one memory device that stores a plurality of instructions which, when executed by the at least one processor, cause the at least one processor to operate with the at least one memory device to:
 acquire a probability distribution forecast and a prior knowledge of the probability distribution forecast; 
 compute an information score based on the probability distribution forecast and the prior knowledge of the probability distribution forecast; and 
 compute a performance score based on the information score. 
   
     
     
         2 . The probability distribution forecast evaluation system of  claim 1 , further comprising instructions that, when executed by the at least one processor, cause the at least one processor to operate with the at least one memory device to:
 acquire a probability distribution realization corresponding to the probability distribution forecast;   compute an accuracy score based on one or more of the probability distribution forecast, the probability distribution realization corresponding to the probability distribution forecast, and the prior knowledge of the probability distribution forecast; and   compute the performance score based on the accuracy score and the information score.   
     
     
         3 . The probability distribution forecast evaluation system of  claim 2 , wherein the accuracy score is computed based on the probability distribution forecast and the probability distribution realization. 
     
     
         4 . The probability distribution forecast evaluation system of  claim 3 , wherein the accuracy score is computed by calculating a dissimilarity score between the probability distribution forecast and the probability distribution realization. 
     
     
         5 . The probability distribution forecast evaluation system of  claim 4 , wherein the dissimilarity score is either (1) the Kullback-Leibler (KL) divergence between the probability distribution forecast and the probability distribution realization, or (2) a quadratic approximation of the KL divergence between the probability distribution forecast and the probability distribution realization. 
     
     
         6 . The probability distribution forecast evaluation system of  claim 2 , wherein the performance score is computed by subtracting the accuracy score from the information score. 
     
     
         7 . The probability distribution forecast evaluation system of  claim 2 , wherein a relative performance score is further computed based on the computed performance score and an entropy of the prior knowledge of the probability distribution forecast. 
     
     
         8 . The probability distribution forecast evaluation system of  claim 1 , wherein the information score is computed by calculating a dissimilarity score between the probability distribution realization and the prior knowledge of the probability distribution forecast. 
     
     
         9 . The probability distribution forecast evaluation system of  claim 8 , wherein the dissimilarity score is either (1) the Kullback-Leibler (KL) divergence between the probability distribution realization and the prior knowledge of the probability distribution forecast, or (2) a quadratic approximation of the KL divergence between the probability distribution realization and the prior knowledge of the probability distribution forecast. 
     
     
         10 . The probability distribution forecast evaluation system of  claim 1 , wherein one or more of the probability distribution forecast and the prior knowledge of the probability distribution forecast are computed based on samples. 
     
     
         11 . The probability distribution forecast evaluation system of  claim 10 , wherein one or more of the probability distribution forecast and the prior knowledge of the probability distribution forecast are partitioned into discrete probability bins. 
     
     
         12 . The probability distribution forecast evaluation system of  claim 10 , wherein one or more of the probability distribution forecast and the prior knowledge of the probability distribution forecast contain sample errors and the performance score is normalized to account for the sample errors. 
     
     
         13 . A method comprising:
 acquiring a probability distribution forecast and a prior knowledge of the probability distribution forecast;   computing an information score based on the probability distribution forecast and the prior knowledge of the probability distribution forecast; and   computing a performance score based on the information score.   
     
     
         14 . The method of  claim 13 , further comprising:
 acquiring a probability distribution realization corresponding to the probability distribution forecast;   computing an accuracy score based on one or more of the probability distribution forecast, the probability distribution realization corresponding to the probability distribution forecast, and the prior knowledge of the probability distribution forecast; and   computing the performance score based on the accuracy score and the information score.   
     
     
         15 . The method of  claim 14 , wherein the accuracy score is computed based on the probability distribution forecast and the probability distribution realization. 
     
     
         16 . The method of  claim 14 , wherein the performance score is computed by subtracting the accuracy score from the information score. 
     
     
         17 . The method of  claim 14 , wherein a relative performance score is further computed based on the computed performance score and an entropy of the prior knowledge of the probability distribution forecast. 
     
     
         18 . The method of  claim 13 , wherein the information score is computed by calculating a dissimilarity score between the probability distribution realization and the prior knowledge of the probability distribution forecast. 
     
     
         19 . The method of  claim 13 , wherein one or more of the probability distribution forecast and the prior knowledge of the probability distribution forecast are computed based on samples, and wherein one or more of the probability distribution forecast and the prior knowledge of the probability distribution forecast are partitioned into discrete probability bins. 
     
     
         20 . The method of  claim 13 , wherein one or more of the probability distribution forecast and the prior knowledge of the probability distribution forecast are computed based on samples, and wherein one or more of the probability distribution forecast and the prior knowledge of the probability distribution forecast contain sample errors and the performance score is normalized to account for the sample errors.