Neural network accelerator

Abstract

Neural network inference may be performed by an apparatus or integrated circuit configured to perform mathematical operations on activation data stored in an activation data memory and weight values stored in a weight memory, to store values resulting from the mathematical operations onto an accumulation memory, to perform activation operations on the values stored in the accumulation memory, to store resulting activation data onto the activation data memory, and to perform inference of a neural network by feeding and synchronizing instructions from an external memory.

What is claimed is: 
     
         1 . An apparatus comprising:
 an activation data memory;   a data loading module configured to load activation data from an external memory onto the activation data memory;   a data storing module configured to store activation data from the activation data memory onto the external memory;   a weight memory;   a weight loading module configured to load weight values from an external memory onto the weight memory;   an accumulation memory;   a plurality of convolution modules configured to perform mathematical operations on the activation data stored in the activation data memory and the weight values stored in the weight memory, and to store values resulting from the mathematical operations onto the accumulation memory;   a plurality of activation modules configured to perform activation operations on the values stored in the accumulation memory, and to store resulting activation data onto the activation data memory; and   an instruction module configured to feed and synchronize instructions from the external memory to the data loading module, the data storing module, the weight loading module, the plurality of convolution modules, and the plurality of activation modules, to perform inference of a convolutional neural network.   
     
     
         2 . An integrated circuit comprising:
 a plurality of convolution modules configured to perform mathematical operations on activation data stored in an activation data memory and weight values stored in a weight memory, and to store values resulting from the mathematical operations onto an accumulation memory;   a plurality of activation modules configured to perform activation operations on the values stored in the accumulation memory, and to store resulting activation data onto the activation data memory; and   an instruction module configured to perform inference of a neural network by feeding and synchronizing instructions from an external memory to the plurality of convolution modules and the plurality of activation modules.   
     
     
         3 . The integrated circuit of  claim 2 , wherein the instruction module is further configured to perform inference of the neural network by causing the plurality of convolution modules to perform the mathematical operations, sequentially by layer, on activation data of corresponding portions of layers of each of a plurality of groups, each group including a number of sequential layers of the neural network. 
     
     
         4 . The integrated circuit of  claim 3 , wherein the instruction module is further configured to perform inference of the neural network by
 loading activation data of corresponding portions in the first layer in each group from the external memory, and storing activation data resulting from the mathematical operations of corresponding portions in the last layer in each group to the external memory.   
     
     
         5 . The integrated circuit of  claim 4 , wherein the instruction module is further configured to perform inference of the neural network by
 clearing activation data of a corresponding portion of a previous layer from the activation data memory in response to storing activation data of a portion of a layer.   
     
     
         6 . The integrated circuit of  claim 5 , wherein the instruction module is further configured to perform inference of the neural network by
 causing the plurality of convolution modules to perform the mathematical operations on activation data of corresponding portions of previous layers stored on the activation data memory by the plurality of activation modules.   
     
     
         7 . The integrated circuit of  claim 6 , wherein the instruction module is further configured to perform inference of corresponding portions of each group by loading activation data from the external memory only once, and storing activation data to the external memory only once. 
     
     
         8 . The integrated circuit of  claim 1 , wherein each activation module is configured to perform activation functions including at least one of Rectified Linear Unit (ReLU), LeakyReLU, Hsigmoid, or H-Swish. 
     
     
         9 . The integrated circuit of  claim 1 , wherein the plurality of convolution modules includes at least one dedicated depth-wise convolution module and at least one point-wise convolution module. 
     
     
         10 . The integrated circuit of  claim 1 , wherein the plurality of convolution modules includes at least one convolution module configured to support combinations of depth-wise convolution and point-wise convolution layers of the neural network. 
     
     
         11 . The computer program of  claim 1 , wherein the neural network is a convolutional neural network, and the portions of each layer are tiles. 
     
     
         12 . The integrated circuit of  claim 1 , wherein the instruction module executes instructions according to an Instruction Set Architecture (ISA). 
     
     
         13 . An integrated circuit comprising:
 an on-chip memory; and   a plurality of logic gates, the plurality of logic gates arranged in groups including
 a plurality of first groups configured to perform mathematical operations on activation data and weight values stored in the on-chip memory, and to store values resulting from the mathematical operations onto the on-chip memory; 
 a plurality of second groups configured to perform activation operations on the resulting values stored in the on-chip memory, and to store resulting activation data onto the on-chip memory; and 
 a third group configured to perform inference of a neural network by feeding and synchronizing instructions from an external memory to the plurality of first groups and the plurality of second groups. 
   
     
     
         14 . The integrated circuit of  claim 13 , wherein the third group is further configured to perform inference of the neural network by causing the plurality of first groups to perform the mathematical operations, sequentially by layer, on activation data of corresponding portions of layers of each of a plurality of groups, each group including a number of sequential layers of the neural network. 
     
     
         15 . The integrated circuit of  claim 14 , wherein the third group is further configured to perform inference of the neural network by
 loading activation data of corresponding portions in the first layer in each group from the external memory, and   storing activation data resulting from the mathematical operations of corresponding portions in the last layer in each group to the external memory.   
     
     
         16 . The integrated circuit of  claim 15 , wherein the third group is further configured to perform inference of the neural network by
 clearing activation data of a corresponding portion of a previous layer from the on-chip memory in response to storing activation data of a portion of a layer.   
     
     
         17 . The integrated circuit of  claim 16 , wherein the third group is further configured to perform inference of the neural network by
 causing the plurality of first groups to perform the mathematical operations on activation data of corresponding portions of previous layers stored on the on-chip memory by the plurality of second groups.   
     
     
         18 . The integrated circuit of  claim 17 , wherein the third group is further configured to perform inference of corresponding portions of each group by loading activation data from the external memory only once, and storing activation data to the external memory only once. 
     
     
         19 . The integrated circuit of  claim 18 , wherein each second group is configured to perform activation functions including at least one of Rectified Linear Unit (ReLU), LeakyReLU, Hsigmoid, or H-Swish. 
     
     
         20 . The integrated circuit of  claim 19 , wherein the plurality of first groups includes at least one first group configured to perform dedicated depth-wise convolution, and at least one first group configured to perform point-wise convolution.