Quantization Algorithms for Analog Hardware Realization of Neural Networks

Abstract

Systems and methods are provided for analog hardware realization of neural networks. The method incudes obtaining a neural network topology and weights of a trained neural network. The method also includes transforming the neural network topology into an equivalent analog network of analog components. The method also includes computing a weight matrix for the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection between analog components of the equivalent analog network. The method also includes generating a schematic model for implementing the equivalent analog network based on the weight matrix, including selecting component parameter values for the analog components.

What is claimed is: 
     
         1 . A method for hardware realization of neural networks, comprising:
 obtaining a neural network topology and weights of a trained neural network;   transforming the neural network topology into an equivalent analog network of analog components including a plurality of operational amplifiers and a plurality of resistors, wherein each operational amplifier represents an analog neuron of the equivalent analog network, and each resistor represents a connection between two analog neurons;   performing resistor quantization and operational amplifier quantization to obtain a quantized network for the equivalent analog network whereby each resistor is assigned a resistance value and each operational amplifier is assigned a value; and   generating, based on the quantized network, a functional behavioral model that includes the plurality of operational amplifiers placed in front-end of line layers (FEOL layers) and the plurality of resistances placed in back-end of line layers (BEOL layers).   
     
     
         2 . The method of  claim 1 , wherein the quantized network is further obtained by performing resistor randomization and operational amplifier randomization. 
     
     
         3 . The method of  claim 1 , wherein performing resistor quantization comprises converting a vector of weights of the trained neural network to particular resistor values of a single analog neuron. 
     
     
         4 . The method of  claim 3 , wherein the resistor values are discrete values from a pre-determined resistor set, each resistor in the resistor set characterized by a respective resistance value and a respective tolerance value. 
     
     
         5 . The method of  claim 3 , wherein the resistor values are limited to a continuous range. 
     
     
         6 . The method of  claim 1 , wherein the resistor quantization includes solving a system of equations and/or inequalities connecting weights of the trained neural network and resistor values, wherein a number of resistors is more than twice a number of weights, and the system of equations has a plurality solutions. 
     
     
         7 . The method of  claim 6 , wherein solving the system of equations includes selecting a solution based on one or more optimization criteria selected from the group consisting of: energy consumption, die area, and minimization of one or more error metrics. 
     
     
         8 . The method of  claim 7 , wherein the one or more optimization criteria include R-quantization error (EQR), which is a mean error of the equivalent analog network with quantized resistors and perfect operational amplifiers versus a math network on a set of input data. 
     
     
         9 . The method of  claim 7 , wherein the one or more optimization criteria include RO-quantization error (EQRO), which is a mean error of the equivalent analog network with quantized resistors and imperfect operational amplifiers with particular pre-defined output model versus a math network on a set of input data. 
     
     
         10 . The method of  claim 7 , wherein the one or more optimization criteria include RO-quantization R-randomization error (EQRO-RR), which is a mean error of the equivalent analog network with quantized resistors with random tolerance error and imperfect operational amplifiers with a predefined output model versus a math network on a set of input data. 
     
     
         11 . The method of  claim 7 , wherein the one or more optimization criteria include RO-quantization RO-randomization error (EQRO-RRO), which is a mean error of T-network with quantized resistors with random tolerance error and imperfect operational amplifiers with a predetermined output model and random tolerance error which affects operational amplifier shift value versus a math network on a set of input data. 
     
     
         12 . The method of  claim 1 , wherein performing operational amplifier quantization includes selecting an appropriate operational amplifier model from a set of predetermined operational amplifier models according to a set of predetermined limitations and/or optimality criteria. 
     
     
         13 . The method of  claim 12 , wherein the set of predetermined limitations and/or optimality criteria includes limitations of operational amplifiers for output currents. 
     
     
         14 . The method of  claim 12 , wherein the set of predetermined limitations and/or optimality criteria includes neuron or operational amplifier die area limitations or minimization. 
     
     
         15 . The method of  claim 12 , wherein the set of predetermined limitations and/or optimality criteria includes neuron or operational amplifier energy consumption limitations or minimization. 
     
     
         16 . The method of  claim 12 , wherein the set of predetermined limitations and/or optimality criteria includes operational amplifier input voltage range limitations. 
     
     
         17 . The method of  claim 12 , wherein the set of predetermined limitations and/or optimality criteria includes minimization of error metrics for resistor quantization. 
     
     
         18 . A system for hardware realization of neural networks, comprising:
 one or more processors; and   memory;   wherein the memory stores one or more programs configured for execution by the one or more processors, and the one or more programs comprising instructions for:   obtaining a neural network topology and weights of a trained neural network;   transforming the neural network topology into an equivalent analog network of analog components including a plurality of operational amplifiers and a plurality of resistors, wherein each operational amplifier represents an analog neuron of the equivalent analog network, and each resistor represents a connection between two analog neurons;   performing resistor quantization and operational amplifier quantization to obtain a quantized network for the equivalent analog network whereby each resistor is assigned a resistance value and each operational amplifier is assigned a value; and   generating, based on the quantized network, a functional behavioral model that includes the plurality of operational amplifiers placed in front-end of line layers (FEOL layers) and the plurality of resistances placed in back-end of line layers (BEOL layers).   
     
     
         19 . The system of  claim 18 , wherein the quantized network is further obtained by performing resistor randomization and operational amplifier randomization. 
     
     
         20 . A non-transitory computer readable storage medium storing one or more programs configured for execution by a computer system having one or more processors, the one or more programs comprising instructions for:
 obtaining a neural network topology and weights of a trained neural network;   transforming the neural network topology into an equivalent analog network of analog components including a plurality of operational amplifiers and a plurality of resistors, wherein each operational amplifier represents an analog neuron of the equivalent analog network, and each resistor represents a connection between two analog neurons;   performing resistor quantization and operational amplifier quantization to obtain a quantized network for the equivalent analog network whereby each resistor is assigned a resistance value and each operational amplifier is assigned a value; and   generating, based on the quantized network, a functional behavioral model that includes the plurality of operational amplifiers placed in front-end of line layers (FEOL layers) and the plurality of resistances placed in back-end of line layers (BEOL layers).