Computer-Implemented Systems and Methods for Testing Large Scale Automatic Forecast Combinations

Abstract

Systems and methods are provided for evaluating performance of forecasting models. A plurality of forecasting models may be generated using a set of in-sample data. Two or more forecasting models from the plurality of forecasting models may be selected for use in generating a combined forecast. An ex-ante combined forecast may be generated for an out-of-sample period using the selected two or more forecasting models. The ex-ante combined forecast may then be compared with a set of actual out-of-sample data to evaluate performance of the combined forecast.

It is claimed: 
     
         1 . A computer-program product tangibly embodied in a non-transitory, machine-readable storage medium having instructions stored thereon, the instructions operable to cause a data processing apparatus to perform operations including:
 accessing an input that represents at least two models that are configured to provide information for ex post forecasts and ex-ante forecasts with regard to a forecast variable;   accessing a historical time series that includes multiple historical observations of the forecast variable;   defining multiple distinct holdout time series within the historical time series, wherein each of the holdout time series includes a portion of the historical observations of the forecast variable and spans a corresponding time period;   generating a combination model by combining the models;   processing multi-step forecasting characteristics of the combination model by using ex-post forecasting at multiple different points of origin, wherein the processing of the multi-step forecasting characteristics of the combination model includes performing operations including:
 using the combination model to generate ex-post forecasted values of the forecast variable at multiple times during the time period; 
 comparing the ex-post forecasted values to coinciding historical observations included in the holdout time series that spans the time period; and 
 calculating ex-post forecast errors based on the comparison; and 
   performing at least one of storing at least part of the multi-step forecasting characteristics of the combination model in a computer data store or transmitting at least part of the multi-step forecasting characteristics of the combination model.   
     
     
         2 . The computer-program product of  claim 1 , wherein:
 the ex-post forecast errors calculated with respect to each of the time periods are displayed with reference to the time periods spanned by the respective holdout time series,   the processing of the multi-step forecasting characteristics of the combination model occurs with respect to each of the time periods, and   the performing at least one of storing at least part of the multi-step forecasting characteristics of the combination model or transmitting at least part of the multi-step forecasting characteristics of the combination model comprises storing the ex-post forecast errors or transmitting the ex-post forecast errors.   
     
     
         3 . The computer-program product of  claim 1 , wherein each of the time periods differs in duration from every other one of the time periods. 
     
     
         4 . The computer-program product of  claim 1 , the instructions are further operable to cause the data processing apparatus to perform operations including:
 displaying an indication of multiple selectable holdout time series durations; and   receiving an input that represents a selection of at least two of the holdout time series durations, wherein the multiple holdout time series are defined based on the selected holdout time series durations.   
     
     
         5 . The computer-program product of  claim 1 , wherein comparing the ex-post forecasted values to coinciding historical observations includes displaying the ex-post forecasted values and the coinciding historical observations on a same graph. 
     
     
         6 . The computer-program product of  claim 5 , the instructions are further operable to cause the data processing apparatus to perform operations including:
 using the combination model to generate an ex-ante forecast of the forecasting variable, wherein the ex-ante forecast includes forecasted values of the forecasting variable with respect to at least one future time or future time period.   
     
     
         7 . The computer-program product of  claim 5 , the instructions are further operable to cause the data processing apparatus to perform operations including:
 receiving an input representing an ex-ante forecasting step-ahead parameter, wherein generating the ex-ante forecast further includes setting a step-ahead time of the ex-ante forecast based on the input.   
     
     
         8 . The computer-program product of  claim 1 , wherein:
 defining multiple holdout time series within the historical time series is performed such that no two of the time periods begin at a same time,   the accessing the input that represents the at least two models is accessed from at least one computer data store, and   the instructions are further operable to cause the data processing apparatus to perform operations including displaying an output from a rolling simulation engine.   
     
     
         9 . A computer-implemented method comprising:
 accessing an input that represents at least two models that are configured to provide information for ex post forecasts and ex-ante forecasts with regard to a forecast variable;   accessing a historical time series that includes multiple historical observations of the forecast variable;   defining multiple distinct holdout time series within the historical time series, wherein each of the holdout time series includes a portion of the historical observations of the forecast variable and spans a corresponding time period;   generating a combination model by combining the models;   processing multi-step forecasting characteristics of the combination model by using ex-post forecasting at multiple different points of origin, wherein the processing of the multi-step forecasting characteristics of the combination model includes performing operations including:
 using the combination model to generate ex-post forecasted values of the forecast variable at multiple times during the time period; 
 comparing the ex-post forecasted values to coinciding historical observations included in the holdout time series that spans the time period; and 
 calculating ex-post forecast errors based on the comparison; and 
   performing at least one of storing at least part of the multi-step forecasting characteristics of the combination model in a computer data store or transmitting at least part of the multi-step forecasting characteristics of the combination model.   
     
     
         10 . The computer-implemented method of  claim 9 , wherein:
 the ex-post forecast errors calculated with respect to each of the time periods are displayed with reference to the time periods spanned by the respective holdout time series,   the processing of the multi-step forecasting characteristics of the combination model occurs with respect to each of the time periods, and   the performing at least one of storing at least part of the multi-step forecasting characteristics of the combination model or transmitting at least part of the multi-step forecasting characteristics of the combination model comprises storing the ex-post forecast errors or transmitting the ex-post forecast errors.   
     
     
         11 . The computer-implemented method of  claim 9 , wherein each of the time periods differs in duration from every other one of the time periods. 
     
     
         12 . The computer-implemented method of  claim 9 , further comprising:
 displaying an indication of multiple selectable holdout time series durations; and   receiving an input that represents a selection of at least two of the holdout time series durations, wherein the multiple holdout time series are defined based on the selected holdout time series durations.   
     
     
         13 . The computer-implemented method of  claim 9 , wherein comparing the ex post forecasted values to coinciding historical observations includes displaying the ex-post forecasted values and the coinciding historical observations on a same graph. 
     
     
         14 . The computer-implemented method of  claim 13 , further comprising:
 using the combination model to generate an ex-ante forecast of the forecasting variable, wherein the ex-ante forecast includes forecasted values of the forecasting variable with respect to at least one future time or future time period.   
     
     
         15 . The computer-implemented method of  claim 13 , further comprising:
 receiving an input representing an ex-ante forecasting step-ahead parameter, wherein generating the ex-ante forecast further includes setting a step-ahead time of the ex-ante forecast based on the input.   
     
     
         16 . The computer-implemented method of  claim 9 , wherein:
 defining multiple holdout time series within the historical time series is performed such that no two of the time periods begin at a same time,   the accessing the input that represents the at least two models is accessed from at least one computer data store, and   the instructions are further operable to cause the data processing apparatus to perform operations including displaying an output from a rolling simulation engine.   
     
     
         17 . A computerized system comprising:
 a processor configured to perform operations including:   accessing an input that represents at least two models that are configured to provide information for ex post forecasts and ex-ante forecasts with regard to a forecast variable;   accessing a historical time series that includes multiple historical observations of the forecast variable;   defining multiple distinct holdout time series within the historical time series, wherein each of the holdout time series includes a portion of the historical observations of the forecast variable and spans a corresponding time period;   generating a combination model by combining the models;   processing multi-step forecasting characteristics of the combination model by using ex-post forecasting at multiple different points of origin, wherein the processing of the multi-step forecasting characteristics of the combination model includes performing operations including:
 using the combination model to generate ex-post forecasted values of the forecast variable at multiple times during the time period; 
 comparing the ex-post forecasted values to coinciding historical observations included in the holdout time series that spans the time period; and 
 calculating ex-post forecast errors based on the comparison; and 
   performing at least one of storing at least part of the multi-step forecasting characteristics of the combination model in a computer data store or transmitting at least part of the multi-step forecasting characteristics of the combination model.   
     
     
         18 . The system of  claim 17 , wherein:
 the ex-post forecast errors calculated with respect to each of the time periods are displayed with reference to the time periods spanned by the respective holdout time series,   the operations for processing of the multi-step forecasting characteristics of the combination model occurs with respect to each of the time periods, and   the operations for performing at least one of storing at least part of the multi-step forecasting characteristics of the combination model or transmitting at least part of the multi-step forecasting characteristics of the combination model comprises storing the ex-post forecast errors or transmitting the ex-post forecast errors.   
     
     
         19 . The system of  claim 17 , wherein each of the time periods differs in duration from every other one of the time periods. 
     
     
         20 . The system of  claim 17 , wherein the operations further include:
 displaying an indication of multiple selectable holdout time series durations; and   receiving an input that represents a selection of at least two of the holdout time series durations, wherein the multiple holdout time series are defined based on the selected holdout time series durations.   
     
     
         21 . The system of  claim 17 , wherein comparing the ex-post forecasted values to coinciding historical observations includes displaying the ex-post forecasted values and the coinciding historical observations on a same graph. 
     
     
         22 . The system of  claim 21 , wherein the operations further include:
 using the combination model to generate an ex-ante forecast of the forecasting variable, wherein the ex-ante forecast includes forecasted values of the forecasting variable with respect to at least one future time or future time period.   
     
     
         23 . The system of  claim 21 , wherein the operations further include:
 receiving an input representing an ex-ante forecasting step-ahead parameter, wherein generating the ex-ante forecast further includes setting a step-ahead time of the ex-ante forecast based on the input.   
     
     
         24 . The system of  claim 17 , wherein:
 defining multiple holdout time series within the historical time series is performed such that no two of the time periods begin at a same time,   the accessing the input that represents the at least two models is accessed from at least one computer data store, and   the instructions are further operable to cause the data processing apparatus to perform operations including displaying an output from a rolling simulation engine.   
     
     
         25 . The system of  claim 17 , further comprising:
 a combined forecast engine that is operable to combine predictions from the models to generate a single combined forecast.   
     
     
         26 . The system of  claim 25 , further comprising:
 a rolling simulation engine that is operable to interact with the combined forecast engine to define the characteristics of the combination model for a rolling simulation analysis.