Compute optimizations for low precision machine learning operations

Abstract

One embodiment provides an apparatus comprising a memory stack including multiple memory dies and a parallel processor including a plurality of multiprocessors. Each multiprocessor has a single instruction, multiple thread (SIMT) architecture, the parallel processor coupled to the memory stack via one or more memory interfaces. At least one multiprocessor comprises a multiply-accumulate circuit to perform multiply-accumulate operations on matrix data in a stage of a neural network implementation to produce a result matrix comprising a plurality of matrix data elements at a first precision, precision tracking logic to evaluate metrics associated with the matrix data elements and indicate if an optimization is to be performed for representing data at a second stage of the neural network implementation, and a numerical transform unit to dynamically perform a numerical transform operation on the matrix data elements based on the indication to produce transformed matrix data elements at a second precision.

What is claimed is: 
     
       1. An apparatus, comprising:
 a memory stack including multiple memory dies; and 
 a parallel processor including a plurality of multiprocessors, each multiprocessor having a single instruction, multiple thread (SIMT) architecture, the parallel processor coupled to the memory stack via one or more memory interfaces, at least one multiprocessor comprising: 
 a multiply-accumulate circuit to perform operations including multiply-accumulate operations on matrix data in a first layer of a neural network implementation to produce a result matrix comprising a plurality of matrix data elements at a first precision, wherein the multiply-accumulate operations include a matrix multiplication operation with values smaller than 32 bits and an accumulate operation having a 32-bit input; 
 precision tracking logic to dynamically evaluate metrics associated with the matrix data elements and, based on the metrics, to indicate if an optimization is to be performed to represent data at a second layer of the neural network implementation, wherein the optimization includes to apply a numerical transform operation to the data to enable representation of the data at the second layer at a second precision; and 
 a numerical transform unit to dynamically perform the numerical transform operation on the matrix data elements, based on the indication, to produce transformed matrix data elements at the second precision, wherein the second precision is lower than the first precision. 
 
     
     
       2. The apparatus of  claim 1 , wherein the numerical transform unit is to multiply or divide the plurality of matrix data elements by a value, based on the indication to perform the optimization, to produce the transformed matrix data elements. 
     
     
       3. The apparatus of  claim 1 , further comprising:
 a compressor to perform compression on the matrix data elements or the transformed matrix data elements prior to writing the matrix data elements or the transformed matrix data elements to one or more memory dies of the memory stack. 
 
     
     
       4. The apparatus of  claim 1 , wherein the parallel processor comprises a graphics processing unit (GPU). 
     
     
       5. The apparatus of  claim 1 , wherein the metrics include a minimum number of bits to store values within the matrix data elements. 
     
     
       6. A graphics processor, comprising:
 a memory stack including multiple memory dies; and 
 a parallel processor including a plurality of multiprocessors, each multiprocessor having a single instruction, multiple thread (SIMT) architecture, the parallel processor coupled to the memory stack via one or more memory interfaces, at least one multiprocessor comprising: 
 a multiply-accumulate circuit to perform operations including multiply-accumulate operations on matrix data in a first layer of a neural network implementation to produce a result matrix comprising a plurality of matrix data elements at a first precision, wherein the multiply-accumulate operations comprise a matrix multiplication operation with values smaller than 32 bits and an accumulate operation having 32-bit inputs; 
 hardware logic to dynamically evaluate metrics associated with the matrix data elements and to indicate, based on the metrics, if an adjustment is to be performed to a representation of data at a second layer of the neural network implementation, wherein the adjustment is to be performed to enable the representation of the data at the second layer of the neural network implementation at a second precision; and 
 numerical transform logic to dynamically perform a numerical transform operation on the matrix data elements based, on the indication, to produce transformed matrix data elements at the second precision, wherein the second precision is lower than the first precision. 
 
     
     
       7. The graphics processor of  claim 6 , wherein the numerical transform logic is to multiply or divide the plurality of matrix data elements by a value, based on the indication to perform the adjustment, to produce the transformed matrix data elements. 
     
     
       8. The graphics processor of  claim 6 , further comprising:
 a compressor to perform compression on the matrix data elements or the transformed matrix data elements prior to writing the matrix data elements or the transformed matrix data elements to one or more memory dies of the memory stack. 
 
     
     
       9. The graphics processor of  claim 6 , wherein each multiprocessor comprises a plurality of general purpose graphics processing unit (GPGPU) cores. 
     
     
       10. The graphics processor of  claim 6 , wherein the metrics include a minimum number of bits to store values of the matrix data elements. 
     
     
       11. A method comprising:
 performing, operations including, via a multiply-accumulate circuit, multiply-accumulate operations, on matrix data in a first layer of a neural network implementation to produce a result matrix comprising a plurality of matrix data elements at a first precision, the multiply-accumulate operations including a matrix multiplication operation with values smaller than 32 bits and an accumulate operation having a 32-bit input, the multiply-accumulate circuit included within at least one multiprocessor of a plurality of multiprocessors, each multiprocessor of the plurality of multiprocessors having a single instruction, multiple thread (SIMT) architecture, the plurality of multiprocessors included within a parallel processor coupled to a memory stack via one or more memory interfaces; 
 dynamically evaluating metrics associated with the matrix data elements via precision tracking logic of the at least one multiprocessor; 
 indicating, via the precision tracking logic and based on the metrics, that an optimization is to be performed for representing data at a second layer of the neural network implementation, wherein the optimization includes applying a numerical transform operation on the data to enable representation of the data at the second layer at a second precision; and 
 dynamically performing the numerical transform operation on the matrix data elements based on the indication, the numerical transform operation performed via a numerical transform unit of the at least one multiprocessor to produce transformed matrix data elements at the second precision, wherein the second precision is lower than the first precision. 
 
     
     
       12. The method of  claim 11 , further comprising, via the numerical transform unit, multiplying or dividing the plurality of matrix data elements by a value, based on the indication to perform the optimization, to produce the transformed matrix data elements. 
     
     
       13. The method of  claim 11 , further comprising:
 compressing the matrix data elements or the transformed matrix data elements prior to writing the matrix data elements or the transformed matrix data elements to one or more memory dies of the memory stack via a compressor of the at least one multiprocessor. 
 
     
     
       14. The method of  claim 11 , wherein the parallel processor comprises a graphics processing unit (GPU). 
     
     
       15. The method of  claim 11 , wherein the metrics include a minimum number of bits to store values of the matrix data elements. 
     
     
       16. A data processing system comprising:
 a memory stack including multiple memory dies; and 
 a parallel processor including a plurality of multiprocessors, each multiprocessor having a single instruction, multiple thread (SIMT) architecture, the parallel processor coupled to the memory stack via one or more memory interfaces, at least one multiprocessor comprising: 
 a multiply-accumulate circuit to perform operations including multiply-accumulate operations on matrix data in a first layer of a neural network implementation to produce a result matrix comprising a plurality of matrix data elements at a first precision, wherein the multiply-accumulate operations include a matrix multiply operation with values smaller than 32 bits and an accumulate operation having a 32-bit input; 
 hardware logic to dynamically evaluate metrics associated with the matrix data elements and to indicate, based on the metrics, if an adjustment is to be performed to a representation of data at a second layer of the neural network implementation, the adjustment performed to enable the representation of the data at the second layer of the neural network implementation at a second precision; and 
 numerical transform logic to dynamically perform a numerical transform operation on the matrix data elements, based on the indication, to produce transformed matrix data elements at the second precision, wherein the second precision is lower than the first precision. 
 
     
     
       17. The data processing system of  claim 16 , wherein the numerical transform logic is to multiply or divide the plurality of matrix data elements by a value, based on the indication to perform the adjustment, to produce the transformed matrix data elements. 
     
     
       18. The data processing system of  claim 16 , further comprising:
 a compressor to perform compression on the matrix data elements or the transformed matrix data elements prior to writing the matrix data elements or the transformed matrix data elements to one or more memory dies of the memory stack. 
 
     
     
       19. The data processing system of  claim 16 , wherein the parallel processor comprises a graphics processing unit (GPU). 
     
     
       20. The data processing system of  claim 16 , wherein metrics include a minimum number of bits to store values of the matrix data elements.