Techniques for deriving and/or leveraging application-centric model metric

Abstract

Techniques for recommending a prediction model from among a number of different prediction models are provided. Each of these prediction models has been trained based on a respective training data set, and each performs in accordance with a respective theoretical performance manifold. An indication of a region definable in relation to the theoretical performance manifolds of the different prediction models is received as input. For each of the different prediction models, the indication of the region is linked to features parameterizing the respective performance manifold; and one or more portions of the respective performance manifold is/are identified based on the features determined by the linking, the portion(s) having a volume and a shape that collectively denote an expected performance of the respective model for the input. The expected performance of the prediction models for the input is compared. Based on the comparison, one or more of the models is/are suggested.

What is claimed is: 
     
         1 . A method of recommending at least one prediction model from among a plurality of different prediction models, each one of the different prediction models having been trained based on a respective training data set, each one of the different prediction models being operable on a set of inputs that at least partially defines a respective theoretical manifold, each of the theoretical manifolds being multidimensional and being parameterized by a plurality of different features, the method comprising:
 receiving an aspect of a region definable in relation to the theoretical manifolds of the different prediction models;   for each of the different prediction models,
 linking the aspect of the region to at least some of the features parameterizing the respective manifold; and 
 identifying one or more portions of the respective manifold based on the features determined by the linking, the one or more portions having a theoretical geometry that denotes an expected manner of performance of the respective model for the aspect of the region; 
   comparing the expected manners of performance of the different prediction models for the aspect of the region; and   based on the comparison, recommending one or more of the different prediction models.   
     
     
         2 . The method of  claim 1 , further comprising, for each of the different prediction models, generating a representation of the respective manifold. 
     
     
         3 . The method of  claim 2 , wherein the generating of the representations of the manifolds comprises, for each of the different prediction models:
 determining which of the plurality of features parameterizing the respective model are strongly correlated with how the respective model performs;   based on the features determined to be strongly correlated with how the respective model performs, creating a plurality of sub-models that together approximate the respective manifold; and   defining the representation of the respective model using the sub-models.   
     
     
         4 . The method of  claim 3 , further comprising, for each of the different prediction models, generating prototype exemplars for each of the created sub-models, the prototype exemplars for each created sub-model being objects to which the respective model can be applied to result in a match with the respective sub-model, the prototype exemplars characterizing the geometry for a portion of the respective manifold. 
     
     
         5 . The method of  claim 4 , wherein the objects are images and/or image collections. 
     
     
         6 . The method of  claim 3 , further comprising determining which features are strongly correlated with how the respective models perform by receiving a user-specified list of one or more features. 
     
     
         7 . The method of  claim 3 , further comprising determining which features are strongly correlated with how the respective models perform by running a residual network feature extractor. 
     
     
         8 . The method of  claim 1 , wherein the training data sets include geospatial and/or geotemporal data. 
     
     
         9 . The method of  claim 1 , wherein the aspect of the region is defined as a set of one or more attributes that parameterize at least one of the different models. 
     
     
         10 . The method of  claim 1 , wherein the aspect of the region is defined using a plurality of images. 
     
     
         11 . A non-transitory computer readable storage medium tangibly storing instructions that, when performed by at least one hardware processor of a computing system, recommend at least one prediction model from among a plurality of different prediction models, each one of the different prediction models having been trained based on a respective training data set, each one of the different prediction being operable on a set of inputs that at least partially defines a respective theoretical manifold, each of the theoretical manifolds being multidimensional and being parameterized by a plurality of different features, the instructions, when performed, causing the computing system to perform operations comprising:
 receiving an aspect of a region definable in relation to the theoretical manifolds of the different prediction models;   for each of the different prediction models,
 linking the aspect of the region to at least some of the features parameterizing the respective manifold; and 
 identifying one or more portions of the respective manifold based on the features determined by the linking, the one or more portions having a theoretical geometry that denotes an expected manner of performance of the respective model for the aspect of the region; 
   comparing the expected manners of performance of the different prediction models for the aspect of the region; and   based on the comparison, recommending one or more of the different prediction models.   
     
     
         12 . The non-transitory computer readable storage medium of  claim 11 , wherein for each of the different prediction models, a representation of the respective manifold is generated. 
     
     
         13 . The non-transitory computer readable storage medium of  claim 12 , wherein the generating of the representations of the manifolds comprises, for each of the different prediction models:
 determining which of the plurality of features parameterizing the respective model are strongly correlated with how the respective model performs;   based on the features determined to be strongly correlated with how the respective model performs, creating a plurality of sub-models that together approximate the respective manifold; and   defining the representation of the respective model using the sub-models.   
     
     
         14 . The non-transitory computer readable storage medium of  claim 12 , wherein for each of the different prediction models, prototype exemplars are generated for each of the created sub-models, the prototype exemplars for each created sub-model being objects to which the respective model can be applied to result in a match with the respective sub-model, the prototype exemplars characterizing the geometry for a portion of the respective manifold. 
     
     
         15 . The non-transitory computer readable storage medium of  claim 12 , wherein the determination of which features are strongly correlated with how the respective models perform is made by running a residual network feature extractor. 
     
     
         16 . A system for recommending at least one prediction model from among a plurality of different prediction models, each one of the different prediction models having been trained based on a respective training data set, each one of the different prediction models being operable on a set of inputs that at least partially defines a respective theoretical manifold, each of the theoretical manifolds being multidimensional and being parameterized by a plurality of different features, the system comprising:
 at least one processor and a memory coupled thereto, the at least one processor being configured to perform operations including at least:
 receive an aspect of a region definable in relation to the theoretical manifolds of the different prediction models; 
 for each of the different prediction models,
 link the aspect of the region to at least some of the features parameterizing the respective manifold; and 
 identify one or more portions of the respective manifold based on the features determined by the linking, the one or more portions having a theoretical geometry that denotes an expected manner of performance of the respective model for the aspect of the region; 
 
 compare the expected manners of performance of the different prediction models for the aspect of the region; and 
 based on the comparison, recommend one or more of the different prediction models. 
   
     
     
         17 . The system of  claim 16 , wherein for each of the different prediction models, a representation of the respective manifold is generated. 
     
     
         18 . The system of  claim 17 , wherein the generating of the representations of the manifolds comprises, for each of the different prediction models:
 determining which of the plurality of features parameterizing the respective model are strongly correlated with how of the respective model performs;   based on the features determined to be strongly correlated with how the respective model performs, creating a plurality of sub-models that together approximate the respective manifold; and   defining the representation of the respective model using the sub-models.   
     
     
         19 . The system of  claim 18 , wherein for each of the different prediction models, prototype exemplars for each of the created sub-models are generated, the prototype exemplars for each created sub-model being objects to which the respective model can be applied to result in a match with the respective sub-model, the prototype exemplars characterizing the geometry for a portion of the respective manifold. 
     
     
         20 . The system of  claim 19 , wherein the objects are images and/or image collections, the objects being parameterized explicitly on the features.