Techniques for deriving and/or leveraging application-centric model metric

Abstract

Techniques for quantifying accuracy of a prediction model that has been trained on a data set parameterized by multiple features are provided. The model performs in accordance with a theoretical performance manifold over an intractable input space in connection with the features. A determination is made as to which of the features are strongly correlated with performance of the model. Based on the features determined to be strongly correlated with performance of the model, parameterized sub-models are created such that, in aggregate, they approximate the intractable input space. Prototype exemplars are generated for each of the created sub-models, with the prototype exemplars for each created sub-model being objects to which the model can be applied to result in a match with the respective sub-model. The accuracy of the model is quantified using the generated prototype exemplars. A recommendation engine is provided for when there are particular areas of interest.

What is claimed is: 
     
         1 . A method of quantifying accuracy of a prediction model that has been trained on a data set parameterized by a plurality of features, the model operating over an input space in connection with the features, the method comprising:
 determining which of the plurality of features are strongly correlated with performance of the model;   based on the features determined to be strongly correlated with performance of the model, creating a plurality of parameterized sub-models that, in aggregate, approximate the input space;   generating prototype exemplars for each of the created sub-models, the prototype exemplars for each created sub-model being objects to which the model can be applied to result in a match with the respective sub-model; and   quantifying the accuracy of the model using the generated prototype exemplars,   wherein the quantifying of the accuracy of the model further comprises, provided that the prototype exemplars are representative of the input space, approximating the variance of the model on a new data set as:
 (a) the sum of a set of one or more performance metrics for the model on each of the prototype exemplars squared multiplied by the probability of the respective prototype exemplar matching its respective sub-model, subtracting (b) the square of the sum of the set of the one or more performance metrics for the model on each of the prototype exemplars, multiplied by the probability of the respective prototype exemplar matching its respective sub-model. 
   
     
     
         2 . The method of  claim 1 , wherein the model is trained to identify objects in images. 
     
     
         3 . The method of  claim 2 , wherein the images are satellite images, and
 wherein the features include spatial extent, National Imagery Interpretability Rating Scale (NIIRS), off-nadir angle, signal-to-noise ratio (SNR), and/or cloud coverage values.   
     
     
         4 . The method of  claim 2 , wherein the quantified accuracy reflects the expected performance of the model identifying an object of a given type from new and/or unseen images. 
     
     
         5 . The method of  claim 1 , wherein the quantifying of the accuracy of the model further comprises, provided that the prototype exemplars are completely parameterized by the features:
 defining the set of one or more performance metrics for the model as a regression on the model for the prototype exemplars; and   (i) approximating the expected performance of the model on the new data set as the sum of the regression on each prototype exemplar multiplied by the probability of the respective prototype exemplar matching its respective sub-model; and/or (ii) approximating the variance of the model on the new data set as (a) the sum of the regression on each prototype exemplar squared multiplied by the probability of the respective prototype exemplar matching its respective sub-model, subtracting (b) the square of the sum of the regression on each prototype exemplar multiplied by the probability of the respective prototype exemplar matching its respective sub-model.   
     
     
         6 . The method of  claim 1 , wherein the input space represents all valid data sets to which the model can be applied. 
     
     
         7 . The method of  claim 1 , wherein the data set on which the prediction model is trained is for a first application, the accuracy of the model is quantified for a data set for a second application, and the first and second applications are different from one another. 
     
     
         8 . The method of  claim 1 , wherein the data set on which the prediction model is trained is for a first geospatial and/or geotemporal image type, the accuracy of the model is quantified for a data set for a second geospatial and/or geotemporal image type, and the first and second geospatial and/or geotemporal image types are different from one another. 
     
     
         9 . A non-transitory computer readable storage medium tangibly storing instructions that, when executed by at least one hardware processor of a computing system, quantify accuracy of a prediction model that has been trained on a data set parameterized by a plurality of features and that operates over an input space in connection with the features, by performing functionality comprising:
 determining which of the plurality of features are strongly correlated with performance of the model;   based on the features determined to be strongly correlated with performance of the model, creating a plurality of parameterized sub-models that, in aggregate, approximate the input space;   generating prototype exemplars for each of the created sub-models, the prototype exemplars for each created sub-model being objects to which the model can be applied to result in a match with the respective sub-model; and   quantifying the accuracy of the model using the generated prototype exemplars,   wherein the quantifying of the accuracy of the model further comprises, provided that the prototype exemplars are representative of the input space, approximating the variance of the model on a new data set as:
 (a) the sum of a set of one or more performance metrics for the model on each of the prototype exemplars squared multiplied by the probability of the respective prototype exemplar matching its respective sub-model, subtracting (b) the square of the sum of the set of the one or more performance metrics for the model on each of the prototype exemplars, multiplied by the probability of the respective prototype exemplar matching its respective sub-model. 
   
     
     
         10 . The non-transitory computer readable storage medium of  claim 9 , wherein the model is trained to identify objects in images. 
     
     
         11 . The non-transitory computer readable storage medium of  claim 10 , wherein the quantified accuracy reflects the expected performance of the model identifying an object of a given type from new and/or unseen images. 
     
     
         12 . The non-transitory computer readable storage medium of  claim 9 , wherein the quantifying of the accuracy of the model further comprises, provided that the prototype exemplars are completely parameterized by the features:
 defining the set of one or more performance metrics for the model as a regression on the model for the prototype exemplars; and   (i) approximating the expected performance of the model on the new data set as the sum of the regression on each prototype exemplar multiplied by the probability of the respective prototype exemplar matching its respective sub-model; and/or (ii) approximating the variance of the model on the new data set as (a) the sum of the regression on each prototype exemplar squared multiplied by the probability of the respective prototype exemplar matching its respective sub-model, subtracting (b) the square of the sum of the regression on each prototype exemplar multiplied by the probability of the respective prototype exemplar matching its respective sub-model.   
     
     
         13 . The non-transitory computer readable storage medium of  claim 9 , wherein the data set on which the prediction model is trained is for a first geospatial and/or geotemporal image type, the accuracy of the model is quantified for a data set for a second geospatial and/or geotemporal image type, and the first and second geospatial and/or geotemporal image types are different from one another. 
     
     
         14 . A system for quantifying accuracy of a prediction model that has been trained on a data set parameterized by a plurality of features, the model operating over an input space in connection with the features, the system comprising:
 an electronic interface over which the model is received; and   processing resources including at least one processor and a memory coupled thereto, the processing resources being configured to at least:
 determine which of the plurality of features are strongly correlated with performance of the model; 
 based on the features determined to be strongly correlated with performance of the model, create a plurality of parameterized sub-models that, in aggregate, approximate the input space; 
 generate prototype exemplars for each of the created sub-models, the prototype exemplars for each created sub-model being objects to which the model can be applied to result in a match with the respective sub-model; and 
 quantify the accuracy of the model using the generated prototype exemplars, 
   wherein the quantifying of the accuracy of the model further comprises, provided that the prototype exemplars are representative of the input space, approximating the variance of the model on a new data set as:
 (a) the sum of a set of one or more performance metrics for the model on each of the prototype exemplars squared multiplied by the probability of the respective prototype exemplar matching its respective sub-model, subtracting (b) the square of the sum of the set of the one or more performance metrics for the model on each of the prototype exemplars, multiplied by the probability of the respective prototype exemplar matching its respective sub-model. 
   
     
     
         15 . The system of  claim 14 , wherein the model is trained to identify objects in images. 
     
     
         16 . The system of  claim 15 , wherein the quantified accuracy reflects the expected performance of the model identifying an object of a given type from new and/or unseen images. 
     
     
         17 . The system of  claim 14 , wherein the processing resources are further configured to at least determine which features are strongly correlated with performance of the model by receiving a user-specified list of one or more features and/or by running a residual network feature extractor. 
     
     
         18 . The system of  claim 14 , wherein the prototype exemplars are generated using synthetics. 
     
     
         19 . The system of  claim 14 , wherein the objects are images and/or image collections, the objects being parameterized explicitly on the features. 
     
     
         20 . The system of  claim 14 , wherein at least one of the features determined to be strongly correlated with part of the model includes a non-linear mapping based on a feature from the data set on which the prediction model is trained.