Deep learning hardware

Abstract

A network of matrix processing units (MPUs) is provided on a device, where each MPU is connected to at least one other MPU in the network, and each MPU is to perform matrix multiplication operations. Computer memory stores tensor data and a master control central processing unit (MCC) is provided on the device to receive an instruction from a host device, where the instruction includes one or more tensor operands based on the tensor data. The MCC invokes a set of operations on one or more of the MPUs based on the instruction, where the set of operations includes operations on the tensor operands. A result is generated from the set of operations, the result embodied as a tensor value.

1 . A integrated circuit (IC) chip comprising:
 a plurality of processing units to collectively perform a matrix multiplication operation with matrix data by performing matrix processing at least partially in parallel, each processing unit of the plurality of processing units to execute an instruction to process a portion of the matrix data to perform a corresponding partial matrix operation;   a plurality of memories, each memory to store the portion of the matrix data to be processed by a corresponding processing unit of the plurality of processing units;   a plurality of pre-multiplication arithmetic engines, each pre-multiplication arithmetic engine to perform a pre-multiplication addition operation with a corresponding portion of the matrix data to generate a new corresponding portion of the matrix data to be used for the matrix multiplication operation;   a plurality of interconnects, a subset of the plurality of interconnects to couple each processing unit of the plurality of processing units to a plurality of neighboring processing units, at least one of the processing units to send partial matrix data to a first neighboring processing unit and to receive partial matrix data from a second neighboring processing unit over corresponding interconnects of the plurality of interconnects;   a first controller, wherein responsive to the first controller, the plurality of processing units are to collectively execute the matrix multiplication operation in accordance with at least one matrix multiplication command or instruction specifying a first input matrix, A, and a second input matrix, B, the plurality of processing units to produce an output matrix, C, by multiplying the first input matrix, A, and the second input matrix, B; and   matrix-wide operation circuitry to perform a matrix-wide operation on the output matrix, C, the matrix-wide operation comprising a max value operation, a min value operation, a sum operation, or a max absolute value operation; and   a plurality of second controllers, each second controller associated with a processing unit of the plurality of processing units, the second controller to retrieve the portion of the matrix data to be processed by a corresponding processing unit of the plurality of processing units from a system memory and to store the portion of the matrix data to a corresponding memory of the plurality of memories.   
     
     
         2 . The IC chip of  claim 1 , wherein the max value operation comprises a max pooling operation. 
     
     
         3 . The IC chip of  claim 2 , wherein the matrix-wide operation is to process data among the elements of the output matrix, C. 
     
     
         4 . The IC chip of  claim 1 , wherein the matrix-wide operation is to process data among the elements of the output matrix, C. 
     
     
         5 . The IC chip of  claim 1 , wherein each pre-multiplication arithmetic engine of the plurality of pre-multiplication arithmetic engines is associated with one of the processing units of the plurality of processing units. 
     
     
         6 . The IC chip of  claim 1 , wherein the first controller comprises a microprocessor. 
     
     
         7 . The IC chip of  claim 1 , wherein the processing units comprise processing clusters. 
     
     
         8 . The IC chip of  claim 7 , further comprising:
 a plurality of local controllers, each local controller to control matrix multiplication operations within a corresponding processing cluster.   
     
     
         9 . The IC chip of  claim 1 , further comprising:
 a memory interface to couple the plurality of memories to a high bandwidth memory (HBM).   
     
     
         10 . The IC chip of  claim 9 , wherein a matrix routine performed by one or more of the plurality of the processing units comprises a distributed matrix multiplication routine, the distributed matrix multiplication routine to be executed by multiple of the plurality of processing units. 
     
     
         11 . The IC chip of  claim 10 , wherein a plurality of instructions of the matrix routine are to be executed to perform one or more convolution operations. 
     
     
         12 . The IC chip of  claim 11 , wherein the matrix routine is associated with an operation in a neural network. 
     
     
         13 . The IC chip of  claim 1 , wherein the matrix multiplication command or instruction is one of a plurality of instructions of a matrix routine, the matrix routine to be performed by one or more of the plurality of the processing units. 
     
     
         14 . The IC chip of  claim 1  further comprising:
 a host interface to couple the plurality of processing units to the first controller.