Analog Hardware Realization of Neural Networks

Abstract

Systems and methods are provided for analog hardware realization of neural networks. The method incudes obtaining a neural network topology and weights of a trained neural network. The method also includes transforming the neural network topology into an equivalent analog network of analog components. The method also includes computing a weight matrix for the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection between analog components of the equivalent analog network. The method also includes generating a schematic model for implementing the equivalent analog network based on the weight matrix, including selecting component parameter values for the analog components.

What is claimed is: 
     
         1 . A method for hardware realization of neural networks, comprising:
 obtaining a neural network topology and weights of a trained neural network;   transforming the neural network topology to an equivalent analog network of analog components, including replacing composition or superposition of linear transformations by a single linear transformation;   computing a weight matrix for the equivalent analog network based on the weights of the trained neural network, wherein each element of the weight matrix represents a respective connection between analog components of the equivalent analog network; and   generating a schematic model for implementing the equivalent analog network based on the weight matrix, including selecting component parameter values for the analog components.   
     
     
         2 . The method of  claim 1 , further comprising:
 prior to transforming the neural network topology to the equivalent analog network:
 adding regularizers to the neural network topology to reduce nominal values of the weights or to a respective reduce weight sum for each neuron; and 
 retraining the trained neural network to obtain updated weights for the weight matrix. 
   
     
     
         3 . The method of  claim 2 , wherein the regularizers include a respective predetermined regularizer for each convolution batch normalization ReLU block, and wherein each predetermined regularizer treats each batch normalization layer as a normalization and calculates combined convolution-batch normalization multipliers applied to an input neural network signal in the signal's propagation path, and reduces the absolute value of combined weights for each neuron. 
     
     
         4 . The method of  claim 1 , wherein transforming the neural network topology into the equivalent analog network comprises translating weights of each batch normalization layer to weights of its previous layer. 
     
     
         5 . The method of  claim 1 , wherein transforming the neural network topology into the equivalent analog network comprises merging layers that do not have an activation function. 
     
     
         6 . The method of  claim 1 , wherein transforming the neural network topology into the equivalent analog network comprises transforming a linear transformation followed by another linear transformation into a single linear transformation. 
     
     
         7 . The method of  claim 1 , wherein transforming the neural network topology into the equivalent analog network comprises transforming layers with ReLU into ReLU1. 
     
     
         8 . The method of  claim 7 , wherein transforming layers with ReLU into ReLU1 comprises maintaining normal operation of the trained neural network during the transformation by analyzing a passage of signals through the trained neural network and performing weight correction. 
     
     
         9 . The method of  claim 8 , wherein performing weight correction comprises:
 adjusting weights so as to restrict signals in the trained neural network below a physical limit.   
     
     
         10 . The method of  claim 8 , wherein performing weight correction comprises:
 when weights of a layer N are divided by a factor, adjusting weights of layer N+1 by multiplying the weights by the factor.   
     
     
         11 . The method of  claim 8 , wherein performing weight correction comprises:
 adjusting weights and bias of one or more neurons and adjustment of weights of outgoing connections of the one or more neurons.   
     
     
         12 . The method of  claim 8 , wherein performing weight correction comprises:
 repeating weight correction for the trained neural network until complete compliance is achieved.   
     
     
         13 . The method of  claim 8 , wherein performing weight correction comprises scaling signals on layers with unlimited ReLU so that they do not exceed a physical limitation. 
     
     
         14 . The method of  claim 1 , wherein transforming the neural network topology to the equivalent analog network comprises introducing additional intermediate layers that limit a number of input or output links of neurons by splitting inputs or outputs of the neurons. 
     
     
         15 . The method of  claim 1 , further comprising:
 pruning at least some connections of the neural network topology.   
     
     
         16 . The method of  claim 1 , further comprising:
 quantizing and/or restricting the weights of the neural network topology.   
     
     
         17 . The method of  claim 1 , further comprising:
 identifying non-linear elements in the neural network topology.   
     
     
         18 . The method of  claim 1 , further comprising (i) calculating a respective range of weights for each layer of the neural network topology and (ii) calculating a respective sum of the weights for each neuron of the neural network topology. 
     
     
         19 . A system for hardware realization of neural networks, comprising:
 one or more processors; and   memory;   wherein the memory stores one or more programs configured for execution by the one or more processors, and the one or more programs comprising instructions for:   obtaining a neural network topology and weights of a trained neural network;   transforming the neural network topology to an equivalent analog network of analog components, including replacing composition or superposition of linear transformations by a single linear transformation;   computing a weight matrix for the equivalent analog network based on the weights of the trained neural network, wherein each element of the weight matrix represents a respective connection between analog components of the equivalent analog network; and   generating a schematic model for implementing the equivalent analog network based on the weight matrix, including selecting component parameter values for the analog components.   
     
     
         20 . A non-transitory computer readable storage medium storing one or more programs configured for execution by a computer system having one or more processors, the one or more programs comprising instructions for:
 obtaining a neural network topology and weights of a trained neural network;   transforming the neural network topology to an equivalent analog network of analog components, including replacing composition or superposition of linear transformations by a single linear transformation;   computing a weight matrix for the equivalent analog network based on the weights of the trained neural network, wherein each element of the weight matrix represents a respective connection between analog components of the equivalent analog network; and   generating a schematic model for implementing the equivalent analog network based on the weight matrix, including selecting component parameter values for the analog components.