Detecting suitability of machine learning models for datasets

Abstract

Apparatuses, systems, program products, and method are disclosed for detecting suitability of machine learning models for datasets. An apparatus includes a training evaluation module configured to calculate a first statistical data signature for a training data set of a machine learning system using one or more predefined statistical algorithms. An apparatus includes an inference evaluation module configured to calculate a second statistical data signature for an inference data set of a machine learning system using one or more predefined statistical algorithms. An apparatus includes a score module configured to calculate a suitability score describing the suitability of a training data set to an inference data set as a function of a first and a second statistical data signature. An apparatus includes an action module configured to perform an action related to a machine learning system in response to a suitability score satisfying an unsuitability threshold.

What is claimed is: 
     
         1 . An apparatus comprising:
 a training evaluation module configured to calculate a first statistical data signature for a training data set of a machine learning system using one or more predefined statistical algorithms, the training data set used to generate a machine learning model;   an inference evaluation module configured to calculate a second statistical data signature for an inference data set of the machine learning system using the one or more predefined statistical algorithms, the inference data set analyzed using the machine learning model;   a score module configured to calculate a suitability score describing the suitability of the training data set to the inference data set as a function of the first and the second statistical data signatures; and   an action module configured to perform an action related to the machine learning system in response to the suitability score satisfying an unsuitability threshold.   
     
     
         2 . The apparatus of  claim 1 , wherein one or more of the training data set and the inference data set comprise feature data sets that are free of labels. 
     
     
         3 . The apparatus of  claim 1 , wherein the score module calculates the suitability score on a per-feature basis based on the training data set and the inference data set. 
     
     
         4 . The apparatus of  claim 3 , wherein:
 the training evaluation module is further configured to:
 determine a probability distribution of the training data set by assigning the values for a feature of the training data set to different groups; 
 determine probabilities for each of the groups of the training data set, the probabilities indicating a likelihood of a value being in the training data set; and 
 calculate an average training probability score for the training data set as a function of the probabilities for the groups; 
   the inference evaluation module is further configured to:
 determine a probability distribution of the inference data set by assigning the values for a feature of the training data set to different groups that correspond to the groups of the training data set; 
 determine probabilities for each of the groups of the inference data set, the probabilities indicating a likelihood of a value of the inference data set being in the training data set; and 
 calculate an average inference probability score based on the probability distribution of the inference data set; and 
   the score module is further configured to calculate the suitability score by normalizing the average inference probability score as a function of the average training probability score.   
     
     
         5 . The apparatus of  claim 1 , wherein the score module calculates the suitability score across a plurality of features of the training data set and the inference data set. 
     
     
         6 . The apparatus of  claim 5 , wherein the training and inference data sets comprise continuous feature data, and:
 the training evaluation module is further configured to:
 generate a Gaussian mixture model as a function of the training data set; 
 determine a likelihood distribution of the training data set based on the generated Gaussian mixture model; and 
 calculate an average training likelihood score based on the likelihood distribution of the training data set; 
   the inference evaluation module is further configured to:
 determine a likelihood distribution of the inference data set based on the generated Gaussian mixture model; and 
 calculate an average inference probability score based on the likelihood distribution of the inference data set; and 
   the score module is further configured to calculate the suitability score by normalizing the average inference likelihood score as a function of the average training likelihood score.   
     
     
         7 . The apparatus of  claim 5 , wherein the training and inference data sets comprise one or more of categorical and binary feature data, and:
 the training evaluation module is further configured to:
 generate a non-random forest of trees as a function of the training data set; 
 determine a probability distribution of the training data set based on the generated non-random forest of trees; and 
 calculate an average training probability score based on the probability distribution of the training data set; 
   the inference evaluation module is further configured to:
 determine a probability distribution of the inference data set as a function of the non-random forest of trees generated based on the training data; and 
 calculate an average inference probability score based on the probability distribution of the inference data set; and 
   the score module is further configured to calculate the suitability score by normalizing the average inference probability score as a function of the average training probability score.   
     
     
         8 . The apparatus of  claim 1 , wherein the inference data set is analyzed using a deep learning machine learning pipeline that comprises a plurality of learning layers, the inference data set selected from a learning layer of the plurality of learning layers that occurs previous to the final learning layer. 
     
     
         9 . The apparatus of  claim 8 , wherein the inference data set is selected from the penultimate learning layer of the plurality of learning layers. 
     
     
         10 . The apparatus of  claim 1 , wherein the action that the action module performs comprises changing the machine learning model to a different machine learning model that is more suitable for the inference data set based on the suitability score. 
     
     
         11 . The apparatus of  claim 1 , wherein the action that the action module performs comprises retraining the machine learning model on a different training data set until the suitability score satisfies a suitability threshold. 
     
     
         12 . The apparatus of  claim 1 , wherein the action that the action module performs comprises sending one or more of an alert, message, and notification that indicates the training data set is unsuitable for the inference data set. 
     
     
         13 . The apparatus of  claim 1 , wherein the action that the action module performs comprises generating one or more labels for the features of the inference data set. 
     
     
         14 . The apparatus of  claim 1 , wherein the machine learning system comprises a plurality of interconnected training and inference pipelines, the suitability score calculated in real-time and on an ongoing basis during machine learning processing to determine the suitability of the training set data to the inference data set at a particular pipeline. 
     
     
         15 . A method comprising:
 calculating a first statistical data signature for a training data set of a machine learning system using one or more predefined statistical algorithms, the training data set used to generate a machine learning model;   calculating a second statistical data signature for an inference data set of the machine learning system using the one or more predefined statistical algorithms, the inference data set analyzed using the machine learning model;   calculating a suitability score describing the suitability of the training data set to the inference data set as a function of the first and the second statistical data signatures; and   performing an action related to the machine learning system in response to the suitability score satisfying an unsuitability threshold.   
     
     
         16 . The method of  claim 15 , wherein one or more of the training data set and the inference data set comprise feature data sets that are free of labels. 
     
     
         17 . The method of  claim 15 , further comprising:
 calculating the suitability score on a per-feature bases by:
 determining a probability distribution of the training data set by assigning the values for a feature of the training data set to different groups; 
 determining probabilities for each of the groups of the training data set, the probabilities indicating a likelihood of a value being in the training data set; 
 calculating an average training probability score for the training data set as a function of the probabilities for the groups; 
 determining a probability distribution of the inference data set by assigning the values for a feature of the training data set to different groups that correspond to the groups of the training data set; 
 determining probabilities for each of the groups of the inference data set, the probabilities indicating a likelihood of a value of the inference data set being in the training data set; 
 calculating an average inference probability score based on the probability distribution of the inference data set; and 
 calculating the suitability score by normalizing the average inference probability score as a function of the average training probability score. 
   
     
     
         18 . The method of  claim 15 , further comprising:
 calculating the suitability score across a plurality of features of continuous feature data of the training data set and the inference data set by:
 generating a Gaussian mixture model as a function of the training data set; 
 determining a likelihood distribution of the training data set based on the generated Gaussian mixture model; 
 calculating an average training likelihood score based on the likelihood distribution of the training data set; 
 determining a likelihood distribution of the inference data set based on the generated Gaussian mixture model; 
 calculating an average inference likelihood score based on the likelihood distribution of the inference data set; and 
 calculating the suitability score by normalizing the average inference likelihood score as a function of the average training likelihood score. 
   
     
     
         19 . The method of  claim 15 , further comprising:
 calculating the suitability score across a plurality of features of one or more of categorical and binary feature data of the training data set and the inference data set by:
 generating a non-random forest of trees as a function of the training data set; 
 determining a probability distribution of the training data set based on the generated non-random forest of trees; 
 calculating an average training probability score based on the probability distribution of the training data set; 
 determining a probability distribution of the inference data set as a function of the non-random forest of trees generated based on the training data set; 
 calculating an average inference probability score based on the probability distribution of the inference data set; and 
 calculating the suitability score by normalizing the average inference probability score as a function of the average training probability score. 
   
     
     
         20 . An apparatus comprising:
 means for calculating a first statistical data signature for a training data set of a machine learning system using one or more predefined statistical algorithms, the training data set used to generate a machine learning model;   means for calculating a second statistical data signature for an inference data set of the machine learning system using the one or more predefined statistical algorithms, the inference data set analyzed using the machine learning model;   means for calculating a suitability score describing the suitability of the training data set to the inference data set as a function of the first and the second statistical data signatures; and   means for performing an action related to the machine learning system in response to the suitability score satisfying an unsuitability threshold.