Optimizing neural network structures for embedded systems

Abstract

A model training and implementation pipeline trains models for individual embedded systems. The pipeline iterates through multiple models and estimates the performance of the models. During a model generation stage, the pipeline translates the description of the model together with the model parameters into an intermediate representation in a language that is compatible with a virtual machine. The intermediate representation is agnostic or independent to the configuration of the target platform. During a model performance estimation stage, the pipeline evaluates the performance of the models without training the models. Based on the analysis of the performance of the untrained models, a subset of models is selected. The selected models are then trained and the performance of the trained models are analyzed. Based on the analysis of the performance of the trained models, a single model is selected for deployment to the target platform.

What is claimed is: 
     
       1. A method for generating a machine-learned model comprising:
 generating an untrained model; 
 generating an intermediate representation of the untrained model, the intermediate representation in an intermediate language compatible with a virtual machine; 
 evaluating the performance of the untrained model, wherein evaluating the performance includes at least one of determining a latency in applying the untrained model in a target system, determining a frequency at which the untrained model can be applied in the target system, determining an amount of resources used by the untrained model, and determining an amount of power consumed by the target system using the untrained model; 
 iteratively generating and evaluating new untrained models, wherein individual new untrained models are generated based on performance of one or more previous untrained models; 
 selecting a subset of models based on the performance of the generated models; 
 training the selected subset of models; 
 evaluating respective accuracies of the subset of models; and 
 selecting a particular model of the subset of models for deployment to the target system based on the accuracies. 
 
     
     
       2. The method of  claim 1  wherein determining an amount of resources used by the model comprises:
 determining a number of floating point operations used by the untrained model when implemented with default kernels. 
 
     
     
       3. The method of  claim 1  wherein determining an amount of resources used by the untrained model comprises:
 determining a number of floating point operations used by the untrained model when implemented with optimized kernels. 
 
     
     
       4. The method of  claim 1  wherein determining an amount of resources used by the untrained model comprises:
 determining a total amount of memory used by the untrained model; and 
 determining a total memory bandwidth used by the untrained model. 
 
     
     
       5. The method of  claim 1  wherein determining an amount of resources used by the untrained model comprises:
 determining an amount of memory used by the untrained model after parameters and variables used by the untrained model have been scheduled; and 
 determining a memory bandwidth used by the untrained model after the parameters and variables used by the untrained model have been scheduled and after operations of the untrained model have been scheduled. 
 
     
     
       6. The method of  claim 1 , further comprising generating an intermediate representation of the subset of the trained models. 
     
     
       7. The method of  claim 1 , wherein selecting the subset of models comprises selecting a first subset of models that perform with at least a specified performance. 
     
     
       8. The method of  claim 4 , further comprising reducing a number of untrained models based on heuristics to identify the models to be trained. 
     
     
       9. A system comprising one or more processors and non-transitory computer readable media storing instructions that when executed by the one or more processors, cause the one or more processors to:
 generate an untrained model; 
 generate an intermediate representation of the untrained model, the intermediate representation in an intermediate language compatible with a virtual machine; 
 evaluate the performance of the untrained model, wherein evaluating the performance includes at least one of determining a latency in applying the untrained model in a target system, determining a frequency at which the untrained model can be applied in the target system, determining an amount of resources used by the untrained model, and determining an amount of power consumed by the target system using the untrained model; 
 iteratively generate and evaluate new untrained models, wherein individual new untrained models are generated based on performance of one or more previous untrained models; 
 select a subset of models based on the performance of the generated models; 
 train the selected subset of models; 
 evaluate respective accuracies of the subset of models; and 
 select a particular model of the subset of models for deployment to the target system based on the accuracies. 
 
     
     
       10. The system of  claim 9 , wherein to determine an amount of resources used by the model, the one or more processors are configured to:
 determine a number of floating point operations used by the untrained model when implemented with default kernels. 
 
     
     
       11. The system of  claim 9 , wherein to determine an amount of resources used by the untrained model, the one or more processors are configured to:
 determine a number of floating point operations used by the untrained model when implemented with optimized kernels. 
 
     
     
       12. The system of  claim 9 , wherein to determine an amount of resources used by the untrained model, the one or more processors are configured to:
 determine a total amount of memory used by the untrained model; and 
 determine a total memory bandwidth used by the untrained model. 
 
     
     
       13. The system of  claim 9 , wherein to determine an amount of resources used by the untrained model, the one or more processors are configured to:
 determine an amount of memory used by the untrained model after parameters and variables used by the untrained model have been scheduled; and 
 determine a memory bandwidth used by the untrained model after the parameters and variables used by the untrained model have been scheduled and after operations of the untrained model have been scheduled. 
 
     
     
       14. The system of  claim 9 , wherein the one or more processors are further configured to generate an intermediate representation of the subset of the models. 
     
     
       15. The system of  claim 9 , wherein to select the subset of models the one or more processors are configured to select a first subset of models that perform with at least a specified performance. 
     
     
       16. The system of  claim 12 , wherein the one or more processors are further configured to reduce a number of untrained models based on heuristics to identify the models to be trained. 
     
     
       17. Non-transitory computer readable media storing instructions that when executed by a system of one or more processors, cause the one or more processors to:
 generate an untrained model; 
 generate an intermediate representation of the untrained model, the intermediate representation in an intermediate language compatible with a virtual machine; 
 evaluate the performance of the untrained model, wherein evaluating the performance includes at least one of determining a latency in applying the untrained model in a target system, determining a frequency at which the untrained model can be applied in the target system, determining an amount of resources used by the untrained model, and determining an amount of power consumed by the target system using the untrained model; 
 iteratively generate and evaluate new untrained models, wherein individual new untrained models are generated based on performance of one or more previous untrained models; 
 select a subset of models based on the performance of the generated models; 
 train the selected subset of models; 
 evaluate respective accuracies of the subset of models; and 
 select a particular model of the subset of models for deployment to the target system based on the accuracies. 
 
     
     
       18. The computer readable media of  claim 17 , wherein to determine an amount of resources used by the model, the one or more processors are configured to:
 determine a number of floating point operations used by the untrained model when implemented with default kernels. 
 
     
     
       19. The computer readable media of  claim 17 , wherein to determine an amount of resources used by the untrained model, the one or more processors are configured to:
 determine a number of floating point operations used by the untrained model when implemented with optimized kernels. 
 
     
     
       20. The computer readable media of  claim 17 , wherein to determine an amount of resources used by the untrained model, the one or more processors are configured to:
 determine a total amount of memory used by the untrained model; and 
 determine a total memory bandwidth used by the untrained model. 
 
     
     
       21. The computer readable media of  claim 17 , wherein to determine an amount of resources used by the untrained model, the one or more processors are configured to:
 determine an amount of memory used by the untrained model after parameters and variables used by the untrained model have been scheduled; and 
 determine a memory bandwidth used by the untrained model after the parameters and variables used by the untrained model have been scheduled and after operations of the untrained model have been scheduled. 
 
     
     
       22. The computer readable media of  claim 17 , wherein the one or more processors are further configured to generate an intermediate representation of the subset of the models. 
     
     
       23. The computer readable media of  claim 17 , wherein to select the subset of models the one or more processors are configured to select a first subset of models that perform with at least a specified performance. 
     
     
       24. The computer readable media of  claim 20 , wherein the one or more processors are further configured to reduce a number of untrained models based on heuristics to identify the models to be trained.