Distributed and redundant machine learning quality management

Abstract

Provided is a process including: writing modelling-object classes using object-oriented modelling of the modelling methods, the modelling-object classes being members of a set of class libraries; writing quality-management classes using object-oriented modelling of quality management, the quality-management classes being members of the set of class libraries; scanning modelling-object classes in the set of class libraries to determine modelling-object class definition information; scanning quality-management classes in the set of class libraries to determine quality-management class definition information; using the modelling-object class definition information and the quality-management class definition information to produce object manipulation functions that allow a quality management system to access methods and attributes of modelling-object classes to manipulate objects of the modelling-object classes; and using the modelling-object class definition information and the quality-management class definition information to produce access to the object manipulation functions.

What is claimed is: 
     
         1 . A tangible, non-transitory, machine-readable medium storing instructions that when executed by one or more processors in a computing system effectuate operations to execute quality management of modelling methods for implementation of a machine learning design in an object-oriented modeling (OOM) framework, the operations comprising:
 writing, with the computing system, modelling-object classes using object-oriented modelling of the modelling methods, the modelling-object classes being members of a set of class libraries;   writing, with the computing system, quality-management classes using object-oriented modelling of quality management, the quality-management classes being members of the set of class libraries;   scanning, with the computing system, modelling-object classes in the set of class libraries to determine modelling-object class definition information;   scanning, with the computing system, quality-management classes in the set of class libraries to determine quality-management class definition information;   using, with the computing system, the modelling-object class definition information and the quality-management class definition information to produce object manipulation functions that allow a quality management system to access methods and attributes of modelling-object classes to manipulate objects of the modelling-object classes; and   using, with the computing system, the modelling-object class definition information and the quality-management class definition information to produce access to the object manipulation functions.   
     
     
         2 . The medium of  claim 1 , wherein:
 executing quality management comprises executing a process that integrates raw data ingestion, manipulation, transformation, composition, and storage for building artificial intelligence models.   
     
     
         3 . The medium of  claim 1 , wherein:
 the modeled quality management comprises management of extract, transform, and load (ETL) phases of a machine learning model designed in the OOM framework.   
     
     
         4 . The medium of  claim 1 , wherein:
 the modeled quality management comprises reporting of model performance of a machine learning model designed in the OOM framework.   
     
     
         5 . The medium of  claim 4 , wherein:
 model performance is measured by recall, precision, or F1 score.   
     
     
         6 . The medium of  claim 1 , wherein:
 the modeled quality management comprises data quality monitoring (DQM).   
     
     
         7 . The medium of  claim 6 , wherein:
 DQM comprises monitoring data sources to detect a new or missing table or data element, data element counts, data element null count and unique counts, or datatype changes.   
     
     
         8 . The medium of  claim 1 , wherein:
 the modeled quality management comprises model quality monitoring (MQM) of a machine learning model designed in the OOM framework.   
     
     
         9 . The medium of  claim 8 , wherein:
 MQM comprises measuring a model-based metric and causing model retraining responsive to detecting more than a threshold amount of drift in the model-based metric.   
     
     
         10 . The medium of  claim 1 , wherein:
 the modeled quality management comprises score quality monitoring (SQM) of a machine learning model designed in the OOM framework.   
     
     
         11 . The medium of  claim 10 , wherein:
 SQM comprises performing a model hypothesis test.   
     
     
         12 . The medium of  claim 10 , wherein:
 SQM comprises computing a lift table or a decile table.   
     
     
         13 . The medium of  claim 1 , wherein:
 the modeled quality management comprises label quality monitoring (LQM) of a machine learning model designed in the OOM framework.   
     
     
         14 . The medium of  claim 13 , wherein:
 LQM comprises determining which data sources among a plurality of data sources are more leverageable or impactful on model performance than other data sources among the plurality of data sources.   
     
     
         15 . The medium of  claim 1 , wherein:
 the modeled quality management comprises bias quality monitoring (BQM) of a machine learning model designed in the OOM framework.   
     
     
         16 . The medium of  claim 15 , wherein
 BQM comprises detecting information bias, selection bias, or confounding by the machine learning model designed in the OOM framework.   
     
     
         17 . The medium of  claim 1 , wherein:
 the modeled quality management comprises privacy quality monitoring (PQM) of a machine learning model designed in the OOM framework.   
     
     
         18 . The medium of  claim 1 , wherein:
 the modeled quality management comprises data quality monitoring (DQM) of a machine learning model designed in the object-oriented modeling (OOM) framework;   DQM comprises monitoring data sources to detect a new or missing table or data element, data element counts, data element null count and unique counts, and datatype changes;   the modeled quality management comprises model quality monitoring (MQM) of the machine learning model designed in the object-oriented modeling (OOM) framework;   MQM comprises measuring a model-based metric and causing model retraining responsive to detecting more than a threshold amount of drift in the model-based metric;   the model-based metric is indicative of an F1 score, accuracy, precision, mean error, media error, distance measure, or recall;   the modeled quality management comprises score quality monitoring (SQM) of the machine learning model designed in the object-oriented modeling (OOM) framework;   SQM comprises performing a model hypothesis test and computing a lift table and a decile table based on predicted probability of positive class membership, based on a cumulative distribution function of positive cases;   the model hypothesis test comprises a Welch's t-test, Kolmogorov-Smirnov test, or a Mann-Whitney U-test;   the modeled quality management comprises label quality monitoring (LQM) of the machine learning model designed in the object-oriented modeling (OOM) framework;   LQM comprises determining which data sources among a plurality of data sources are more leverageable or impactful on model performance than other data sources among the plurality of data sources;   the modeled quality management comprises bias quality monitoring (BQM) of the machine learning model designed in the object-oriented modeling (OOM) framework;   BQM comprises detecting information bias, selection bias, and confounding by the machine learning model designed in the object-oriented modeling (OOM) framework;   the modeled quality management comprises privacy quality monitoring (PQM) of the machine learning model designed in the OOM framework.   
     
     
         19 . The medium of  claim 1 , wherein:
 the modeled quality management comprises a process to determine data source reliability.   
     
     
         20 . The medium of  claim 1 , wherein:
 an attribute of a quality-management object in one of the quality-management classes comprise means for characterizing quality with the attribute of the quality-management object.