Backpressure for Accelerated Deep Learning

Abstract

Techniques in advanced deep learning provide improvements in one or more of accuracy, performance, and energy efficiency. An array of processing elements performs flow-based computations on wavelets of data. Each processing element comprises a respective compute element and a respective routing element. Each compute element comprises virtual input queues. Each router enables communication via wavelets with at least nearest neighbors in a 2D mesh. Routing is controlled by respective virtual channel specifiers in each wavelet and routing configuration information in each router. Each router comprises data queues. The virtual input queues of the compute element and the data queues of the router are managed in accordance with the virtual channels. Backpressure information, per each of the virtual channels, is generated, communicated, and used to prevent overrun of the virtual input queues and the data queues.

1 . (canceled) 
     
     
         2 . A method comprising:
 managing a plurality of virtual input queues of a processing element having a coupling to a fabric, each virtual input queue enabled to store at most a respective number of fabric packets received via the fabric, the coupling associated with a plurality of fabric virtual channels each associated with one of the virtual input queues;   managing a plurality of backpressure indicators, each backpressure indicator associated with a respective one of the fabric virtual channels and enabled to selectively indicate one of a stall state and a ready state, each backpressure indicator set to indicate the ready state responsive to the virtual input queue associated with the fabric virtual channel associated with the respective backpressure indicator holding less than a respective threshold number of fabric packets and otherwise set to indicate the stall state; and   transmitting the backpressure indicators via the coupling and through routing paths enabled by the fabric and determinable at least in part by referencing information identified by fabric virtual channel identifiers respectively associated with each of the fabric virtual channels and wherein each of the fabric packets comprises a respective instance of one of the fabric virtual channel identifiers.   
     
     
         3 . The method of  claim 2 , wherein the processing element is a first processing element, the coupling is a first coupling to the fabric and further comprising receiving the backpressure indicators via the fabric at a second processing element, the second processing element comprising a second coupling to the fabric, the second coupling associated with the fabric virtual channels. 
     
     
         4 . The method of  claim 3 , further comprising in the second processing element and responsive to ones of the received backpressure indicators indicating the stall state, stalling transmission associated with the fabric virtual channels associated with the ones of the received backpressure indicators indicating the stall state, the stalling transmission being with respect to fabric packets from the second processing element destined for the first processing element. 
     
     
         5 . The method of  claim 3 , further comprising in the second processing element and responsive to the receiving, selecting one or more fabric packets for transmission from the second processing element to the first processing element via the fabric, wherein the selected fabric packets are associated with one of the fabric virtual channels and the one of the fabric virtual channels is associated with a one of the received backpressure indicators indicating the ready state. 
     
     
         6 . The method of  claim 3 , wherein the first processing element and the second processing element are fabricated via wafer-scale integration. 
     
     
         7 . The method of  claim 6 , wherein the wafer-scale integration is implemented using a significant portion of a wafer as an element by leaving the wafer substantially whole. 
     
     
         8 . The method of  claim 2 , wherein at least one of the fabric virtual channels is used for communicating at least one of data and control associated with one or more of: computing an activation of a neural network, computing a partial sum of activations of a neural network, computing an error of a neural network, computing a gradient estimate of a neural network, and updating a weight of a neural network. 
     
     
         9 . The method of  claim 8 , wherein one or more of the fabric packets comprise a data element that comprises at least a portion of one or more of: a weight of a neural network, an activation of a neural network, a partial sum of activations of a neural network, an error of a neural network, a gradient estimate of a neural network, and a weight update of a neural network. 
     
     
         10 . An apparatus comprising:
 a fabric;   a processing element comprising
 a plurality of virtual input queues, 
 virtual input queue control hardware logic circuitry, 
 a coupling to the fabric, 
 a plurality of backpressure indicators, 
 backpressure indicator control hardware logic circuitry, and 
 transmission hardware logic circuitry; and 
   wherein
 each virtual input queue is enabled to store at most a respective number of fabric packets received via the fabric, 
 the virtual input queue control hardware logic circuitry is enabled to manage the virtual input queues, 
 the coupling is associated with a plurality of fabric virtual channels each associated with one of the virtual input queues, 
 each backpressure indicator is associated with a respective one of the fabric virtual channels and enabled to selectively indicate one of a stall state and a ready state, 
 the backpressure indicator control hardware logic circuitry is enabled to manage the backpressure indicators via setting respective ones of the backpressure indicators to indicate the ready state responsive to the virtual input queue associated with the fabric virtual channel associated with the respective backpressure indicator holding less than a respective threshold number of fabric packets and otherwise setting the respective backpressure indicator to indicate the stall state, 
 the transmission hardware logic circuitry is enabled to transmit the backpressure indicators via the coupling and through routing paths enabled by the fabric and determinable at least in part by referencing information identified by fabric virtual channel identifiers respectively associated with each of the fabric virtual channels, and 
 each of the fabric packets comprises a respective instance of one of the fabric virtual channel identifiers. 
   
     
     
         11 . The apparatus of  claim 10 , wherein the processing element is a first processing element, the coupling is a first coupling to the fabric; further comprising a second processing element comprising a second coupling to the fabric and reception hardware logic circuitry; and wherein the second coupling is associated with the fabric virtual channels and the reception hardware logic circuitry is enabled to receive the backpressure indicators via the fabric at the second processing element. 
     
     
         12 . The apparatus of  claim 11 , wherein the second processing element further comprises stalling hardware logic circuitry enabled, responsive to ones of the received backpressure indicators indicating the stall state, to stall transmission associated with the fabric virtual channels associated with the ones of the received backpressure indicators indicating the stall state, the stalling transmission being with respect to fabric packets from the second processing element destined for the first processing element. 
     
     
         13 . The apparatus of  claim 12 , wherein the second processing element further comprises selection hardware logic circuitry enabled, responsive to the stalling hardware logic circuitry, to select one or more fabric packets for transmission from the second processing element to the first processing element via the fabric, wherein the selected fabric packets are associated with one of the fabric virtual channels and the one of the fabric virtual channels is associated with a one of the received backpressure indicators indicating the ready state. 
     
     
         14 . The apparatus of  claim 11 , wherein the first processing element and the second processing element are fabricated via wafer-scale integration that is implemented using a significant portion of a wafer as an element by leaving the wafer substantially whole. 
     
     
         15 . The apparatus of  claim 10 , wherein at least one of the fabric virtual channels is used for communicating at least one of data and control associated with one or more of: computing an activation of a neural network, computing a partial sum of activations of a neural network, computing an error of a neural network, computing a gradient estimate of a neural network, and updating a weight of a neural network; and further wherein one or more of the fabric packets comprise a data element that comprises at least a portion of one or more of: a weight of a neural network, an activation of a neural network, a partial sum of activations of a neural network, an error of a neural network, a gradient estimate of a neural network, and a weight update of a neural network. 
     
     
         16 . A system comprising:
 means for managing a plurality of virtual input queues of a processing element having a coupling to a fabric, each virtual input queue enabled to store at most a respective number of fabric packets received via the fabric, the coupling associated with a plurality of fabric virtual channels each associated with one of the virtual input queues;   means for managing a plurality of backpressure indicators, each backpressure indicator associated with a respective one of the fabric virtual channels and enabled to selectively indicate one of a stall state and a ready state, each backpressure indicator set to indicate the ready state responsive to the virtual input queue associated with the fabric virtual channel associated with the respective backpressure indicator holding less than a respective threshold number of fabric packets and otherwise set to indicate the stall state; and   means for transmitting the backpressure indicators via the coupling and through routing paths enabled by the fabric and determinable at least in part by referencing information identified by fabric virtual channel identifiers respectively associated with each of the fabric virtual channels and wherein each of the fabric packets comprises a respective instance of one of the fabric virtual channel identifiers.   
     
     
         17 . The system of  claim 16 , wherein the processing element is a first processing element, the coupling is a first coupling to the fabric and further comprising means for receiving the backpressure indicators via the fabric at a second processing element, the second processing element comprising a second coupling to the fabric, the second coupling associated with the fabric virtual channels. 
     
     
         18 . The system of  claim 17 , further comprising means for, responsive to ones of the received backpressure indicators indicating the stall state, stalling transmission associated with the fabric virtual channels associated with the ones of the received backpressure indicators indicating the stall state, the stalling transmission being with respect to fabric packets from the second processing element destined for the first processing element. 
     
     
         19 . The system of  claim 17 , further comprising means for, responsive to the means for receiving, selecting one or more fabric packets for transmission from the second processing element to the first processing element via the fabric, wherein the selected fabric packets are associated with one of the fabric virtual channels and the one of the fabric virtual channels is associated with a one of the received backpressure indicators indicating the ready state. 
     
     
         20 . The system of  claim 17 , wherein the first processing element and the second processing element are fabricated via wafer-scale integration that is implemented using a significant portion of a wafer as an element by leaving the wafer substantially whole. 
     
     
         21 . The system of  claim 16 , wherein at least one of the fabric virtual channels is used for communicating at least one of data and control associated with one or more of: computing an activation of a neural network, computing a partial sum of activations of a neural network, computing an error of a neural network, computing a gradient estimate of a neural network, and updating a weight of a neural network; and further wherein one or more of the fabric packets comprise a data element that comprises at least a portion of one or more of: a weight of a neural network, an activation of a neural network, a partial sum of activations of a neural network, an error of a neural network, a gradient estimate of a neural network, and a weight update of a neural network.