Automated versioning and evaluation of machine learning workflows

Abstract

A method comprising determining a plurality of configurations for a machine learning workflow characterizing a pinna to predict a head related transfer function defining transformation of sound by the pinna. Each of the configurations comprises parameters for constructing and executing the machine learning workflow, a program code to be used for executing the machine learning workflow, and machine learning models to be used for tasks in the machine learning workflow. The method comprises generating a plurality of identifiers for the plurality of configurations, respectively, based on the machine learning models and the parameters by normalizing and encoding. The method comprises executing the machine learning workflow in accordance with the selected one of the configurations, analyzing results of the execution of the machine learning workflow to generate a metric, and storing results of the execution of the machine learning workflow along with the one of the identifiers and the metric.

What is claimed is: 
     
         1 . A method comprising:
 determining a plurality of configurations for a machine learning workflow characterizing a pinna to predict a head related transfer function defining transformation of sound by the pinna, each of the configurations comprising parameters for constructing and executing the machine learning workflow, a program code to be used for executing the machine learning workflow, and machine learning models to be used for tasks in the machine learning workflow;   generating a plurality of identifiers for the plurality of configurations, respectively, based on the machine learning models and the parameters by:
 normalizing one or more parameters of a selected one of the configurations, wherein the normalizing comprises converting the one or more parameters to a data structure suitable for hashing, wherein the converting comprises one or more of organizing and formatting the one or more parameters, and wherein the organizing and formatting comprises one or more of removing a parameter, sorting the one or more parameters, and removing one or more characters from the one or more parameters; and 
 encoding the normalized one or more parameters using a hashing algorithm to generate one of the identifiers that is unique to the one or more parameters of the selected one of the configurations; 
   executing the machine learning workflow in accordance with the selected one of the configurations;   analyzing results of the execution of the machine learning workflow to generate a metric; and   storing results of the execution of the machine learning workflow along with the one of the identifiers and the metric.   
     
     
         2 . The method of  claim 1  further comprising:
 generating the metric for each of the configurations; 
 identifying one of the configurations with the best metric; and 
 selecting a program code of the identified one of the configurations for deployment. 
 
     
     
         3 . The method of  claim 2  further comprising identifying a best performing combination of parameters using a brute force search and the metrics. 
     
     
         4 . The method of  claim 2  further comprising identifying a best performing combination of parameters using a genetic algorithm and the metrics. 
     
     
         5 . The method of  claim 2  further comprising identifying a best performing combination of the parameters using mutation and the metrics. 
     
     
         6 . The method of  claim 2  further comprising:
 selecting parameters for a first task from a first one of the configurations and selecting parameters for a second task from a second one of the configurations, wherein the selections are based on corresponding metrics; and 
 identifying a best performing combination of parameters using the metrics. 
 
     
     
         7 . The method of  claim 1  further comprising:
 selecting a next one of the configurations; 
 determining, based on the stored results, if the machine learning workflow is already executed in accordance with the next one of the configurations; 
 executing the machine learning workflow in accordance with the next one of the configurations if the determining indicates that the machine learning workflow is not executed in accordance with the next one of the configurations; and 
 not executing the machine learning workflow in accordance with the next one of the configurations if the determining indicates that the machine learning workflow is already executed in accordance with the next one of the configurations. 
 
     
     
         8 . A non-transitory, computer-readable medium storing instructions which when executed by a processor configure the processor to:
 determine a plurality of configurations for a machine learning workflow characterizing a pinna to predict a head related transfer function defining transformation of sound by the pinna, each of the configurations comprising parameters for constructing and executing the machine learning workflow, a program code to be used for executing the machine learning workflow, and machine learning models to be used for tasks in the machine learning workflow;   generate a plurality of identifiers for the plurality of configurations, respectively, based on the machine learning models and the parameters by:
 normalizing one or more parameters of a selected one of the configurations, wherein the normalizing comprises converting the one or more parameters to a data structure suitable for hashing, wherein the converting comprises one or more of organizing and formatting the one or more parameters, and wherein the organizing and formatting comprises one or more of removing a parameter, sorting the one or more parameters, and removing one or more characters from the one or more parameters; and 
 encoding the normalized one or more parameters using a hashing algorithm to generate one of the identifiers that is unique to the one or more parameters of the selected one of the configurations; 
   execute the machine learning workflow in accordance with the selected one of the configurations;   analyze results of the execution of the machine learning workflow to generate a metric; and   store results of the execution of the machine learning workflow along with the one of the identifiers and the metric.   
     
     
         9 . The non-transitory, computer-readable medium of  claim 8  wherein the instructions further configure the processor to:
 generate the metric for each of the configurations; 
 identify one of the configurations with the best metric; and 
 select a program code of the identified one of the configurations for deployment. 
 
     
     
         10 . The non-transitory, computer-readable medium of  claim 8  wherein the instructions further configure the processor to identify a best performing combination of parameters using a brute force search and the metrics. 
     
     
         11 . The non-transitory, computer-readable medium of  claim 8  wherein the instructions further configure the processor to identify a best performing combination of parameters using a genetic algorithm and the metrics. 
     
     
         12 . The non-transitory, computer-readable medium of  claim 8  wherein the instructions further configure the processor to identify a best performing combination of the parameters using mutation and the metrics. 
     
     
         13 . The non-transitory, computer-readable medium of  claim 8  wherein the instructions further configure the processor to:
 select parameters for a first task from a first one of the configurations and selecting parameters for a second task from a second one of the configurations, wherein the selections are based on corresponding metrics; and 
 identify a best performing combination of parameters using the metrics. 
 
     
     
         14 . The non-transitory, computer-readable medium of  claim 8  wherein the instructions further configure the processor to:
 select a next one of the configurations; 
 determine, based on the stored results, if the machine learning workflow is already executed in accordance with the next one of the configurations; 
 execute the machine learning workflow in accordance with the next one of the configurations if the machine learning workflow is not executed in accordance with the next one of the configurations; and 
 not execute the machine learning workflow in accordance with the next one of the configurations if the machine learning workflow is already executed in accordance with the next one of the configurations.