Deep learning inference efficiency technology with early exit and speculative execution

Abstract

Systems, apparatuses and methods may provide for technology that processes an inference workload in a first subset of layers of a neural network that prevents or inhibits data dependent branch operations, conducts an exit determination as to whether an output of the first subset of layers satisfies one or more exit criteria, and selectively bypasses processing of the output in a second subset of layers of the neural network based on the exit determination. The technology may also speculatively initiate the processing of the output in the second subset of layers while the exit determination is pending. Additionally, when the inference workloads include a plurality of batches, the technology may mask one or more of the plurality of batches from processing in the second subset of layers.

We claim: 
     
       1. A semiconductor apparatus comprising:
 one or more substrates; and 
 logic coupled to the one or more substrates, wherein the logic is implemented at least partly in one or more of configurable logic or fixed-functionality hardware logic, the logic coupled to the one or more substrates to: 
 execute a first subset of layers of a neural network on a common execution unit, wherein the neural network prevents data dependent branch operations; 
 process an inference workload in the first subset of layers; 
 conduct an exit determination as to whether an output of the first subset of layers satisfies one or more exit criteria; 
 initiate an operation to process the output in a second subset of layers of the neural network while the exit determination is pending; 
 clear a loop iteration counter associated with the second subset of layers if the output satisfies the one or more exit criteria; and 
 set the loop iteration counter if the output does not satisfy the one or more exit criteria. 
 
     
     
       2. The semiconductor apparatus of  claim 1 , wherein the operation is a speculative operation. 
     
     
       3. The semiconductor apparatus of  claim 1 , wherein the inference workload is to include a plurality of batches and the logic coupled to the one or more substrates is to mask one or more of the plurality of batches from processing in the second subset of layers. 
     
     
       4. The semiconductor apparatus of  claim 1 , wherein the one or more exit criteria is to include a ratio of data is relatively far away from a decision boundary to data that is relatively close to the decision boundary. 
     
     
       5. A computing system comprising:
 a network controller to obtain an inference workload; and 
 a processor including one or more substrates and logic coupled to the one or more substrates, wherein the logic is implemented at least partly in one or more of configurable logic or fixed-functionality hardware logic, the logic coupled to the one or more substrates to:
 process the inference workload in a first subset of layers of a neural network that prevents data dependent branch operations, 
 conduct an exit determination as to whether an output of the first subset of layers satisfies one or more exit criteria, 
 initiate an operation to process the output in a second subset of layers of the neural network while the exit determination is pending, and 
 selectively bypass processing of the output in the second subset of layers of the neural network based on the exit determination. 
 
 
     
     
       6. The computing system of  claim 5 , wherein the logic coupled to the one or more substrates is to clear a loop iteration counter associated with the second subset of layers if the output satisfies the one or more exit criteria. 
     
     
       7. The computing system of  claim 6 , wherein the logic coupled to the one or more substrates is to set the loop iteration counter if the output does not satisfy the one or more exit criteria. 
     
     
       8. The computing system of  claim 5 , wherein the operation is a speculative operation. 
     
     
       9. The computing system of  claim 5 , wherein the inference workload is to include a plurality of batches and the logic coupled to the one or more substrates is to mask one or more of the plurality of batches from processing in the second subset of layers. 
     
     
       10. A semiconductor apparatus comprising:
 one or more substrates; and 
 logic coupled to the one or more substrates, wherein the logic is implemented at least partly in one or more of configurable logic or fixed-functionality hardware logic, the logic coupled to the one or more substrates to: 
 process an inference workload in a first subset of layers of a neural network that prevents data dependent branch operations; 
 conduct an exit determination as to whether an output of the first subset of layers satisfies one or more exit criteria; 
 initiate an operation to process the output in a second subset of layers of the neural network while the exit determination is pending; and 
 selectively bypass processing of the output in a second subset of layers of the neural network based on the exit determination. 
 
     
     
       11. The semiconductor apparatus of  claim 10 , wherein the logic coupled to the one or more substrates is to clear a loop iteration counter associated with the second subset of layers if the output satisfies the one or more exit criteria. 
     
     
       12. The semiconductor apparatus of  claim 11 , wherein the logic coupled to the one or more substrates is to set the loop iteration counter if the output does not satisfy the one or more exit criteria. 
     
     
       13. The semiconductor apparatus of  claim 10 , wherein the operation is a speculative operation. 
     
     
       14. The semiconductor apparatus of  claim 10 , wherein the inference workload is to include a plurality of batches and the logic coupled to the one or more substrates is to mask one or more of the plurality of batches from processing in the second subset of layers. 
     
     
       15. The semiconductor apparatus of  claim 10 , wherein the one or more exit criteria is to include a ratio of data is relatively far away from a decision boundary to data that is relatively close to the decision boundary. 
     
     
       16. The semiconductor apparatus of  claim 10 , wherein the logic coupled to the one or more substrates is to execute the first subset of layers on a common execution unit. 
     
     
       17. The semiconductor apparatus of  claim 10 , wherein the logic coupled to the one or more substrates includes transistor channel regions that are positioned within the one or more substrates. 
     
     
       18. A method comprising:
 processing an inference workload in a first subset of layers of a neural network that prevents data dependent branch operations; 
 conducting an exit determination as to whether an output of the first subset of layers satisfies one or more exit criteria; 
 initiating an operation to process the output in a second subset of layers of the neural network while the exit determination is pending; and 
 selectively bypassing processing of the output in a second subset of layers of the neural network based on the exit determination. 
 
     
     
       19. The method of  claim 18 , wherein selectively bypassing the processing of the output in the second subset of layers includes clearing a loop iteration counter associated with the second subset of layers if the output satisfies the one or more exit criteria. 
     
     
       20. The method of  claim 18 , wherein selectively bypassing the processing of the output in the second subset of layers includes setting the loop iteration counter if the output does not satisfy the one or more exit criteria. 
     
     
       21. The method of  claim 18 , wherein the operation is a speculative operation. 
     
     
       22. The method of  claim 18 , wherein the inference workload includes a plurality of batches and the method further comprises selectively bypassing processing of the output in the second subset of layers includes masking one or more of the plurality of batches from processing in the second subset of layers. 
     
     
       23. The method of  claim 18 , wherein the one or more exit criteria includes a ratio of data that is relatively far away from a decision boundary to data that is relatively close to the decision boundary. 
     
     
       24. The method of  claim 18 , further including executing the first subset of layers on a common execution unit.