Methods of operating an artificial neural network using a compute-in-memory accelerator and a bitwise activation function

Abstract

A method includes providing an artificial neural network comprising a compute-in-memory accelerator, the artificial neural network further comprising a hidden layer including a first plurality of artificial neurons; and training the artificial neural network using a bitwise modified rectified linear unit activation function for ones of the first plurality of artificial neurons, the bitwise modified rectified linear unit activation function comprising a bit activation function, which is configured to generate an output that is proportional to an input when the input is less than a critical threshold and configured to generate an output that is independent of the input when the input is greater than the critical threshold, wherein the input comprises a sum, across a second plurality of artificial neurons of a preceding layer of the artificial neural network having a plurality of weights associated therewith, respectively, of a product of an output from a respective one of the second plurality of artificial neurons and one bit of a respective one of the plurality of weights.

What is claimed is: 
     
       1. A method, comprising:
 providing an artificial neural network comprising a compute-in-memory accelerator, the artificial neural network further comprising a hidden layer including a first plurality of artificial neurons; and 
 training the artificial neural network using a bitwise modified rectified linear unit activation function for ones of the first plurality of artificial neurons, the bitwise modified rectified linear unit activation function comprising a bit activation function, which is configured to generate an output that is proportional to an input when the input is less than a critical threshold and configured to generate an output that is independent of the input when the input is greater than the critical threshold, 
 wherein the input comprises a sum, across a second plurality of artificial neurons of a preceding layer of the artificial neural network having a plurality of weights associated therewith, respectively, of a product of an output from a respective one of the second plurality of artificial neurons and one bit of a respective one of the plurality of weights, and 
 wherein generating the sum comprises charge sharing between a plurality of output lines corresponding to respective bit positions of the plurality of weights; 
 wherein training the artificial neural network comprises: 
 adjusting, based on outputs of the artificial neural network generated in response to a training data set, one or more parameters of the compute-in-memory accelerator, wherein the one or more parameters comprise: a voltage pulse width associated with the output from the respective one of the second plurality of artificial neurons, a plurality of loading capacitances associated with the plurality of output lines respectively, or a combination thereof. 
 
     
     
       2. The method of  claim 1 , wherein respective ones of the plurality of weights comprises a plurality of bits, and
 wherein the bitwise modified rectified linear unit activation function comprises a summation across the plurality of bits of a product of a value of a respective one of the plurality of bits and the output of the bit activation function. 
 
     
     
       3. The method of  claim 2 , wherein the bit activation function is configured to generate a constant output that is independent of the input for input values greater than the critical threshold. 
     
     
       4. The method of  claim 3 , wherein the constant output is an inverse of a scaling factor for the plurality of weights. 
     
     
       5. The method of  claim 1 , wherein the compute-in-memory accelerator comprises an analog-to-digital converter coupled to the plurality of output lines, the method further comprising: determining an output voltage (Vout_low) for each of the plurality of output lines when a total number of pulses based on the output from the respective one of the second plurality of artificial neurons and applied to a respective one of the output lines equals the critical threshold; and setting a smallest reference voltage of the analog-to-digital converter approximately equal to Vout_low. 
     
     
       6. The method of  claim 5 , wherein the analog-to-digital converter is configured to convert voltage values less than Vout_low to a maximum output. 
     
     
       7. The method of  claim 1 , wherein the plurality of outputs from the second plurality of artificial neurons, respectively, and the plurality of weights are non-negative. 
     
     
       8. The method of  claim 1 , further comprising:
 operating the artificial neural network in an inference mode responsive to training the artificial neural network, 
 wherein the artificial neural network is configured to generate an output inference responsive to a current data set when in the inference mode. 
 
     
     
       9. A method, comprising:
 providing an artificial neural network comprising a compute-in-memory accelerator, the artificial neural network further comprising a hidden layer including a first plurality of artificial neurons; and 
 training the artificial neural network using a bitwise modified rectified linear unit activation function for ones of the first plurality of artificial neurons, the bitwise modified rectified linear unit activation function comprising a bit activation function, which is configured to generate an output that is proportional to an input when the input is less than a critical threshold and configured to generate an output that is independent of the input when the input is greater than the critical threshold, and wherein the input is generated by charge sharing between a plurality of output lines corresponding to respective bit positions of a plurality of weights, the plurality of weights being associated with a second plurality of artificial neurons of a preceding layer of the artificial neural network; 
 wherein training the artificial neural network comprises adjusting a value of the critical threshold based on outputs of the artificial neural network generated in response to a training data set and adjusting one or more parameters of the compute-in-memory accelerator based on the critical threshold, wherein the one or more parameters comprise a voltage pulse width associated with an output from a respective one of the second plurality of artificial neurons of a preceding layer of the artificial neural network, and a plurality of loading capacitances associated with the plurality of output lines respectively. 
 
     
     
       10. The method of  claim 9 , wherein the compute-in-memory accelerator comprises an analog-to-digital converter coupled to the output lines, the method further comprising: determining an output voltage (Vout_low) for each of the plurality of output lines when a total number of pulses based on the output from the respective one of the second plurality of artificial neurons and applied to a respective one of the output lines equals the critical threshold; and setting a smallest reference voltage of the analog-to-digital converter approximately equal to Vout_low. 
     
     
       11. The method of  claim 9 , wherein the input comprises a sum, across the second plurality of artificial neurons of the preceding layer of the artificial neural network having the plurality of weights associated therewith, respectively, of a product of an output from a respective one of the second plurality of artificial neurons and one bit of a respective one of the plurality of weights. 
     
     
       12. A method, comprising:
 providing an artificial neural network comprising a compute-in-memory accelerator, the artificial neural network further comprising a hidden layer including a first plurality of artificial neurons, the compute-in-memory accelerator comprising an analog-to-digital converter coupled to a plurality of output lines respectively corresponding to each bit position of ones of a plurality of weights associated with a second plurality of artificial neurons of a preceding layer of the artificial neural network, respectively; 
 training the artificial neural network using a bitwise modified rectified linear unit activation function for ones of the first plurality of artificial neurons, the bitwise modified rectified linear unit activation function comprising a bit activation function, which is configured to generate an output that is proportional to an input when the input is less than a critical threshold and configured to generate an output that is independent of the input when the input is greater than the critical threshold; 
 determining an output voltage (Vout_low) for each of the plurality of output lines when a total number of pulses applied to a respective one of the output lines equals the critical threshold; and setting a smallest reference voltage of the analog-to-digital converter approximately equal to Vout_low, 
 wherein the input is generated by charge sharing between a plurality of output lines corresponding to respective bit positions of the plurality of weights; 
 adjusting one or more parameters of the compute-in-memory accelerator based on the critical threshold, wherein the one or more parameters comprise a voltage pulse width associated with an output from a respective one of the second plurality of artificial neurons of the preceding layer of the artificial neural network, and a plurality of loading capacitances associated with the plurality of output lines respectively. 
 
     
     
       13. The method of  claim 12 , wherein the analog-to-digital converter is configured to convert voltage values less than Vout_low to a maximum output. 
     
     
       14. The method of  claim 12 , wherein the input comprises a sum, across the second plurality of artificial neurons, respectively, of a product of an output from a respective one of the second plurality of artificial neurons and one bit of a respective one of the plurality of weights.