Data Processing in a Machine Learning Computer

Abstract

A computer-implemented method of training a multi-layer neural network comprising a set of network weights, comprising: processing the training data in respective forward and backward passes through multiple layers, the forward pass comprising computing a set of activations in dependence on the network weights and training data, and the backward pass comprising: computing gradients of a pre-determined loss function with respect to the network weights and/or activations, wherein an adjustment parameter is applied to at least a subset of values in the neural network, the values comprising at least one of: the network weights, the activations, the gradients with respect to activations and the gradients with respect to weights; updating the network weights in dependence on the computed gradients; computing a proportion of the subset of values falling above a predefined threshold; and updating the adjustment parameter in dependence on the computed proportion.

1 . A computer-implemented method of training, based on a set of training data, a multi-layer neural network comprising a set of network weights, the method comprising:
 processing the training data in respective forward and backward passes through a sequence of layers of the network, the forward pass comprising computing a set of activations by applying an activation function in dependence on the network weights and training data, and the backward pass comprising:
 computing gradients of a pre-determined loss function with respect to the network weights and/or 
 computing gradients of the pre-determined loss function with respect to the computed activations of the network, 
   wherein an adjustment parameter is applied to at least a subset of values in the neural network, the values comprising at least one of:
 the network weights, 
 the activations computed in the forward pass, 
 the gradients with respect to activations computed in the backward pass, and 
 the gradients with respect to weights computed in the backward pass; 
   updating the network weights in dependence on the computed gradients with respect to the weights;   computing a proportion of the subset of values falling above a predefined threshold; and   updating the adjustment parameter applied to the subset of machine learning parameters in dependence on the computed proportion.   
     
     
         2 . The method of  claim 1 , wherein the adjustment parameter is a scale factor, and wherein the scale factor is applied on the backward pass to at least a subset of the gradients with respect to at least one of the activations and the gradients with respect to the network weights, wherein the scale factor is updated in dependence on the proportion of the gradients of that subset that have a value falling above a pre-defined threshold. 
     
     
         3 . The method of  claim 2 , comprising applying the scale factor to at least one of gradients with respect to weights and gradients with respect to activations of all layers of the network by multiplying the loss function by the scale factor. 
     
     
         4 . The method of  claim 2 , comprising constructing a histogram of gradients, the histogram comprising a plurality of bins, wherein the scale factor is updated based on a proportion of gradients occupying bins above a threshold value. 
     
     
         5 . The method of  claim 4 , comprising constructing a respective histogram of gradients for each layer of the neural network, wherein the proportion of gradients occupying each of a set of bins for each histogram is input to an accumulator to obtain an aggregated proportion for each bin, the scale factor being derived by computing an aggregated proportion occupying bins above an overall threshold. 
     
     
         6 . The method of  claim 4 , comprising constructing a respective histogram of gradients for each layer, wherein for each layer a respective layer-wise scale factor is applied during the backward pass, the layer-wise scale factor being updated based on a proportion of gradients in the histogram for the corresponding layer occupying bins above a corresponding layer-wise threshold value. 
     
     
         7 . The method of  claim 4  when implemented on a plurality of processors, wherein each processor processes a respective subset of the training data in each of the forward and backward passes, and computes a respective histogram of gradients for the corresponding subset of the training data, each histogram having defined a common set of bins, wherein the proportion of gradients occupying each bin of the set of bins defined for each histogram is aggregated to obtain an aggregated proportion for each bin, with a scale factor being derived by computing an aggregated proportion occupying bins above an overall threshold. 
     
     
         8 . The method of  claim 1 , comprising storing at least a subset of the network weights, gradients and activations in computer memory in floating-point format. 
     
     
         9 . The method of  claim 8 , comprising storing at least a subset of the network weights, gradients and activations in computer memory in eight-bit floating-point format. 
     
     
         10 . The method of  claim 8 , comprising storing at least a subset of the network weights, gradients and activations in computer memory in sixteen-bit floating-point format. 
     
     
         11 . The method of  claim 8 , comprising storing the subset of values in a floating-point format, and wherein the adjustment parameter is an exponent bias applied to the floating-point representations of the subset of weights, gradients and activations. 
     
     
         12 . The method of  claim 11 , wherein the subset of values in the neural network is a subset of network weights and activations and the adjustment parameter is an exponent bias applied to the subset of values of the network weights and activations in the forward pass. 
     
     
         13 . The method of  claim 11 , wherein a subset of network weights, activations and gradients which are inputs to compute operations in at least one of the forward and backward passes are stored in eight-bit floating-point format, the compute operations comprising at least one of a matrix operation and a convolution operation. 
     
     
         14 . A computer system comprising one or more processors configured to train a multi-layer neural network comprising a set of network weights, and memory holding the network weights, the processor configured to train the neural network by:
 receiving a set of training data;   processing the training data in respective forward and backward passes through a sequence of layers of the network, the forward pass comprising computing a set of activations by applying an activation function in dependence on the network weights and training data, and the backward pass comprising determining a set of gradients of a pre-determined loss function with respect to the weights and/or activations of the network, wherein an adjustment parameter is applied to at least a subset of values in the neural network, wherein the values on the forward pass comprise at least one of the network weights and computed activations, and the values on the backwards pass comprise the computed gradients with respect to activations and gradients with respect to weights;   storing the values to memory;   updating the network weights in dependence on the computed gradients with respect to the weights;   on at least one of the forward and backward pass, computing a proportion of the subset of values falling above a predefined threshold; and   updating the adjustment parameter applied to the subset of machine learning parameters in dependence on the computed proportion.   
     
     
         15 . The computer system of  claim 14 , comprising a plurality of processors, wherein each processor is configured to process a respective subset of the training data. 
     
     
         16 . The computer system of  claim 15 , wherein the adjustment parameter is updated in dependence on an aggregated proportion of values for all processors falling above a predefined threshold, the aggregated proportion computed by aggregating a computed proportion of the subset of values falling above the predefined threshold for each of the plurality of processors. 
     
     
         17 . A non-transitory computer-readable storage medium storing computer program instructions which when executed perform a method of training, based on a set of training data, a multi-layer neural network comprising a set of network weights, the method comprising:
 processing the training data in respective forward and backward passes through a sequence of layers of the network, the forward pass comprising computing a set of activations by applying an activation function in dependence on the network weights and training data, and the backward pass comprising determining a set of gradients of a pre-determined loss function with respect to the weights and/or activations of the network, wherein an adjustment parameter is applied to at least a subset of values in the neural network, and wherein the values on the forward pass comprise at least one of the network weights and computed activations, and the values on the backwards pass comprise the computed gradients with respect to activations and gradients with respect to weights;   updating the network weights in dependence on the computed gradients with respect to the weights;   on at least one of the forward and backward pass, computing a proportion of the subset of values falling above a predefined threshold; and   updating the adjustment parameter applied to the subset of machine learning parameters in dependence on the computed proportion.