Neural Network Processor

Abstract

A circuit for performing neural network computations for a neural network comprising a plurality of neural network layers, the circuit comprising: a matrix computation unit configured to, for each of the plurality of neural network layers: receive a plurality of weight inputs and a plurality of activation inputs for the neural network layer, and generate a plurality of accumulated values based on the plurality of weight inputs and the plurality of activation inputs; and a vector computation unit communicatively coupled to the matrix computation unit and configured to, for each of the plurality of neural network layers: apply an activation function to each accumulated value generated by the matrix computation unit to generate a plurality of activated values for the neural network layer.

What is claimed is: 
     
         1 . (canceled) 
     
     
         2 . A circuit for performing neural network computations for a neural network comprising a plurality of neural network layers, the circuit comprising:
 a matrix computation unit configured to, for each of the plurality of neural network layers:
 pre-load a plurality of weight inputs for the neural network layer into cells of the matrix computation unit; 
 after pre-loading the plurality of weight inputs, shift activations for the neural network layer through the cells of the matrix computation unit along a first dimension of the matrix computation unit; 
 compute, at each of the cells, one or more multiplications to generate a respective product for each of the one or more multiplications, each multiplication being between a weight input pre-loaded into the cell and an activation value shifted to the cell; and 
 shift, along a second dimension of the matrix computation unit, partial sums that are generated based on the respective products for each of the one or more multiplications. 
   
     
     
         3 . The circuit of  claim 2 , wherein the matrix computation unit is configured to:
 generate a vector of outputs comprising multiple accumulated values, each of the multiple accumulated values corresponding to a partial sum that is generated based on a respective product of at least one multiplication between a weight input pre-loaded into a cell of the matrix computation unit and an activation value shifted to the cell.   
     
     
         4 . The circuit of  claim 3 , comprising:
 a vector unit coupled to the matrix computation unit, the vector unit being configured to, for each of the plurality of neural network layers:
 receive the vector of outputs from the matrix computation unit; 
 apply an activation function to each of the multiple accumulated values in the vector of outputs; and 
 generate a plurality of activated values for the neural network layer based on the activation function that is applied to each of the multiple accumulated values. 
   
     
     
         5 . The circuit of  claim 2 , wherein the matrix computation unit is configured to:
 pre-load at least a subset of the plurality of weight inputs into the cells of the matrix computation unit as stationary weight inputs to be multiplied with two or more different activations that are shifted through the cells of the matrix computation unit; and   store the stationary weight inputs in respective weight registers disposed in the cells of the matrix computation unit.   
     
     
         6 . The circuit of  claim 5 , wherein the matrix computation unit is configured to:
 generate a plurality of accumulated values corresponding to dot products of multiplications between a plurality of stationary weight inputs and different activation values that are shifted through the cells of the matrix computation unit along the first dimension of the matrix computation unit.   
     
     
         7 . The circuit of  claim 2 , wherein:
 a partial sum that is shifted through the cells of the matrix computation unit corresponds to a portion of accumulated values used to generate a vector of accumulated values as an output of the neural network layer; and   the matrix computation unit is configured to provide the vector of accumulated values to a vector unit coupled to the matrix computation unit.   
     
     
         8 . The circuit of  claim 7 , wherein:
 the vector unit is configured to generate a vector of activation values using the vector of accumulated values; and   the circuit is configured to shift the vector of activation values to the cells of the matrix computation unit as an input to a second, different neural network layer.   
     
     
         9 . The circuit of  claim 2 , comprising a weight interface coupled to the matrix computation unit, the weight interface being configured to:
 pre-load, along the second dimension of the matrix computation unit, the plurality of weight inputs as stationary weight inputs by storing the weight inputs in weight registers disposed in the cells of the matrix computation unit.   
     
     
         10 . The circuit of  claim 9 , comprising a unified buffer coupled to the matrix computation unit, the unified buffer being configured to:
 provide, along the first dimension of the matrix computation unit, the activations for the neural network layer to the cells of the matrix computation unit after the plurality of weight inputs are pre-loaded in the matrix computation unit as stationary weight inputs.   
     
     
         11 . The circuit of  claim 10 , wherein the first dimension of the matrix computation unit and the second dimension of the matrix computation unit are the same dimension. 
     
     
         12 . A method for performing neural network computations for a neural network comprising a plurality of neural network layers using a circuit comprising a matrix computation unit, wherein the method comprises, for each of the plurality of neural network layers:
 pre-loading a plurality of weight inputs for the neural network layer into cells of the matrix computation unit;   after pre-loading the plurality of weight inputs, shifting activations for the layer through the cells of the matrix computation unit along a first dimension;   computing, at each of the cells, one or more multiplications to generate a respective product for each of the one or more multiplications, each multiplication being between a weight input pre-loaded into the cell and a different activation value shifted to the cell; and   shifting, along a second dimension of the matrix computation unit, partial sums that are generated based on the respective products for each of the one or more multiplications.   
     
     
         13 . The method of  claim 12 , comprising:
 generating a vector of outputs that includes multiple accumulated values, each of the multiple accumulated values corresponding to a partial sum that is generated based on a respective product of at least one multiplication between a weight input pre-loaded into a cell of the matrix computation unit and an activation value shifted to the cell.   
     
     
         14 . The method of  claim 13 , wherein the circuit comprises a vector unit coupled to the matrix computation unit, and the method comprises:
 receiving, by the vector unit, the vector of outputs from the matrix computation unit;   applying, by the vector unit, an activation function to each of the multiple accumulated values in the vector of outputs; and   generating, by the vector unit, a plurality of activated values for the neural network layer based on the activation function that is applied to each of the multiple accumulated values.   
     
     
         15 . The method of  claim 12 , comprising:
 pre-loading at least a subset of the plurality of weight inputs into the cells of the matrix computation unit as stationary weight inputs to be multiplied with two or more different activations that are shifted through the cells of the matrix computation unit; and   storing the stationary weight inputs in respective weight registers disposed in the cells of the matrix computation unit.   
     
     
         16 . The method of  claim 15 , comprising:
 generating a plurality of accumulated values corresponding to dot products of multiplications between a plurality of stationary weight inputs and different activation values that are shifted through the cells of the matrix computation unit along the first dimension of the matrix computation unit.   
     
     
         17 . The method of  claim 12 , wherein a partial sum that is shifted through the cells of the matrix computation unit corresponds to a portion of accumulated values used to generate a vector of accumulated values as an output of the neural network layer, and the method comprises:
 providing the vector of accumulated values to a vector unit coupled to the matrix computation unit;   generating, by the vector unit, a vector of activation values using the vector of accumulated values; and   shifting, by the circuit, the vector of activation values to the cells of the matrix computation unit as an input to a second, different neural network layer.   
     
     
         18 . The method of  claim 12 , wherein the circuit comprises a weight interface coupled to the matrix computation unit, and the method comprises:
 pre-loading, using the weight interface and along the second dimension of the matrix computation unit, the plurality of weight inputs as stationary weight inputs,
 comprising storing the weight inputs in weight registers disposed in the cells of the matrix computation unit. 
   
     
     
         19 . The method of  claim 18 , wherein the circuit comprises a unified buffer coupled to the matrix computation unit, and the method comprises:
 providing, using the unified buffer and along the first dimension of the matrix computation unit, the activations for the neural network layer to the cells of the matrix computation unit after the plurality of weight inputs are pre-loaded in the matrix computation unit as stationary weight inputs.   
     
     
         20 . The method of  claim 19 , wherein the first dimension of the matrix computation unit and the second dimension of the matrix computation unit are the same dimension.