Analog learning engine and method

Abstract

A neural network learning mechanism has a device which perturbs analog neurons to measure an error which results from perturbations at different points within the neural network and modifies weights and biases to converge to a target.

What is claimed is: 
     
         1 . A neural network error contour generation mechanism comprising a device which perturbs analog neurons to measure an error which results from perturbations at different points within the neural network. 
     
     
         2 . The neural network error contour generation mechanism of  claim 1 , comprising
 a neuron summer to integrate a perturbation; and   analog circuit sampling and holding an activation result of the perturbation to calculate σ′(z).   
     
     
         3 . The neural network error contour generation mechanism of  claim 2 , comprising multiplier circuit multiplying σ′(z) by a curl of a cost function to generate output layer errors. 
     
     
         4 . The neural network error contour generation mechanism of  claim 3 , wherein the neurons are comprised of on or more of switched charge multiplier, division and current mode summations, decision or similar circuits. 
     
     
         5 . The neural network error contour generation mechanism of  claim 3 , comprising means for generating errors for layers below an output layer by using one of switched charge multiplier or division and current or charge domain summations to generate remaining error values working backwards from the output layer errors. 
     
     
         6 . The neural network error contour generation mechanism of  claim 1 , wherein the error caused by a perturbation at each weighted input to a neuron is measured at a respective output of the neural network. 
     
     
         7 . The neural network error contour generation mechanism of  claim 1 , comprising a neural network update circuit modifying analog weight values to direct the neural network error contour generation mechanism towards a target in response to an error contour generated. 
     
     
         8 . The neural network error contour generation mechanism of  claim 6 , comprising a neural network update circuit modifying analog weight values to direct the neural network error contour generation mechanism towards a target in response to an error contour generated. 
     
     
         9 . The neural network error contour generation mechanism of  claim 1 , wherein an error contour generated is stored in an analog memory. 
     
     
         10 . The neural network error contour generation mechanism of  claim 8 , wherein the error contour generated is stored in an analog memory. 
     
     
         11 . The neural network error contour generation mechanism of  claim 1 , comprising a circuit modifying analog bias values to direct the neural network error contour generation mechanism towards a target. 
     
     
         12 . The neural network error contour generation mechanism of  claim 1 , wherein the error is a quadratic difference. 
     
     
         13 . A backpropagation mechanism comprising:
 a neural network error contour generation mechanism comprising a device which perturbs analog neurons to measure an error which results from perturbations at different points within the neural network;   wherein a set of m mini-batches of n training samples are inputted to the neural network error contour generation mechanism, wherein n is the number of training examples and m the number of mini-batches of training examples;   wherein each weight in the neural network error is modified in part according to an average over n samples of the curl of an error function multiplied by a local activation derivative to move towards a target.   
     
     
         14 . A backpropagation means comprising:
 an error contour comprising a change in a neural network resulting from perturbation of each weighted input sum;   wherein a set of m mini-batches of n training samples are inputted to the neural network, wherein n is the number of training examples and m the number of mini-batches of training examples;   wherein each weight in the neural network is modified according to an average over n samples of the curl of an error function multiplied by a local activation derivative to move towards a target.   
     
     
         15 . A weight tuning circuit comprising:
 one of a gated current or charge input representing an input value to match;   a ΣΔ modulator using two switched charge reservoirs in inverter configuration;   wherein an output of the ΣΔ modulator adjusts currents sources feeding a node between the two switched charge reservoirs against a comparator reference voltage.   
     
     
         16 . The weight tuning circuit of  claim 15 , wherein multiple weights use the comparator reference voltage, the comparator reference voltage adjusted to eliminate offset errors, 
     
     
         17 . A weight tuning means comprising:
 one of a gated current or charge input representing an input value to match;   a ΣΔ modulator using two switched charge reservoirs in inverter configuration;   wherein a current representing a weight is set from anode between the two switched charge reservoirs; and   wherein a resulting integrated value is compared to a comparator reference to generate an average accurate value over multiple cycles.   
     
     
         18 . The weight tuning circuit of  claim 17 , wherein multiple weights use the comparator reference voltage, the comparator reference voltage adjusted to eliminate offset errors. 
     
     
         19 . A learning device whereby an incremental charge is propagated through a neural network to determine its sensitivity on the output of the neural network and where those weights which produce a significant change towards a target are modified by a controller to minimize said error. 
     
     
         20 . The learning device of  19  where said modifications to said weights are made by said controller in conformance with the fourth rule of backpropagation by adjustment of said weight current source magnitude. 
     
     
         21 . A charge domain switched charge multiply and sum circuit, accepting weighted charge inputs, where the limited pulse duration of the charge transfer modifies the noise bandwidth compared to a switch capacitor circuit such that the noise is dramatically reduced and thus required capacitance dramatically reduced. 
     
     
         22 . A charge domain neuron, comprising the switch charge multiply and sum circuit of  21  and further comprising a charge domain ReLU decision circuit wherein a weighted input charge is introduced to a weighted summer and a threshold level for a ReLU circuit is loaded as charge into a second weighted summer and the outputs pulses of said summer circuits are simultaneously OR'ed to produce a ReLU output. 
     
     
         23 . An input operand converting device by which to convert a pulse input value to a current source input value in a charge domain memory cell comprising:
 a charge domain memory circuit which accepts a pulse input which gates a current input so as to store a weighted charge on a summing node;   a controller which applies a maximal current input value to said memory circuit while accepting a pulse input value so as to generate said weighted charge on said summing node;   a delay lock loop;   wherein said delay lock loop accepts the output pulse of said memory circuit and then the controller applies a maximal input pulse after which said delay lock loop adjusts the magnitude of said current weight value until the output of said memory circuit matches the pulse with initially introduced.   
     
     
         24 . A backpropagation mechanism consisting of:
 a controller;   the charge domain neurons of  claim 22 ;   the input operand converting device of  claim 23 ;   wherein the four equations of back propagation are implemented by utilizing combinations of said charge domain neurons and operand converting devices;   wherein said controller adjusts weights and biases in conformance with the fourth rule of backpropagation.   
     
     
         25 . An oversampled neural network where the inputs to neurons in the neural network are oversampled and where each comprises a ΣΔ modulator wherein the Δ of each input operand versus a quantizer output value is filtered with gain and then reapplied to said quantizer such that accurate values of the neural network operation are the average of said oversampled values.