Systems and methods for implementing operational transformations for restricted computations of a mixed-signal integrated circuit

Abstract

Systems and methods for improving a computational performance of a mixed-signal integrated circuit includes identifying a suboptimal graph component of a computation graph of a subject application, wherein: (i) the computation graph comprises a plurality of graphical nodes representing computational operations and a plurality of graphical edges representing data dependencies between the graphical nodes, and (ii) the suboptimal graph component comprises a subset of the plurality of graphical nodes and the plurality of graphical edges that do not satisfy an optimal operation threshold; at compile time, selectively applying an optimizing transformation to the suboptimal graph component based on attributes of a first activation function within the suboptimal graph component, wherein the optimization transformation, when applied, transforms the suboptimal graph component to an optimal graph component that satisfies the optimal operation threshold; and reconstructing the computation graph using the optimal graph component in a place of the suboptimal graph component.

What is claimed is: 
     
         1 . A method for transforming one or more components of a computation network graph, the method comprising:
 identifying an input leading into an activation operation node of a computation graph, the input comprising a dataset having negative data values;   implementing one or more computing devices executing a compiler program that transforms the activation operation node of the computation graph based on the identifying the input having the dataset having the negative data values, the transforming includes:   replacing a first activation function of the activation operation node with a second activation function, wherein:
 (a) the first activation function, if applied at runtime to the dataset having the negative data values, replaces the negative data values with zero, and 
 (b) the second activation function, when applied to the dataset having the negative data values, shifts the negative data values into a positive range of values. 
   
     
     
         2 . The method according to  claim 1 , further comprising:
 identifying a computation operation of the computation graph that outputs the dataset having the negative data values, wherein the transforming further includes:
 adjusting the weights bias value of the computation operation based on a bit range of an analog-to-digital converter (ADC). 
   
     
     
         3 . The method according to  claim 2 , wherein
 the adjusting the weights bias value of the computation operation includes reducing the weights bias value by one-half the bit range of the ADC.   
     
     
         4 . The method according to  claim 1 , wherein
 the first activation function comprises a rectifier linear unit,   the second activation function comprises a hard sigmoid, and   after the transformation, the activation operation node comprises the hard sigmoid.   
     
     
         5 . The method according to  claim 1 , wherein
 at runtime, the activation operation node comprising the second activation function is applied following a first computation operation of the computation graph that outputs the dataset having the negative data values.   
     
     
         6 . The method according to  claim 5 , wherein
 the computation operation is implemented by a matrix multiply accelerator.   
     
     
         7 . The method according to  claim 5 , wherein
 the computation operation is implemented by a streaming arithmetic logic unit.   
     
     
         8 . The method according to  claim 2 , wherein
 the adjusting the weights bias value of the computation operation is further based on one or more of a gain and a scaling factor.   
     
     
         9 . A method for transforming one or more components of a computation network graph, the method comprising:
 identifying a first computation operation in a first branch of a computation graph, the first computation operation having a saturating activation function;   identifying a second computation operation in a second branch of the computation graph, the second computation operation having the saturating activation function;   at compile time, transforming the first computation operation and the second computation operation, wherein the transforming includes:
 fusing the first computation operation and the second computation operation into a combined computation operation. 
   
     
     
         10 . The method according to  claim 9 , wherein
 the first computation operation is implemented by a first matrix multiply accelerator; and   the second computation operation is implemented by a second matrix multiply accelerator.   
     
     
         11 . The method according to  claim 9 , wherein
 the saturating activation function is applied to an output of the combined computation operation.   
     
     
         12 . The method according to  claim 9 , wherein
 the saturating activation function of the first computation operation and the second computation operation comprises a hardtanh activation.   
     
     
         13 . The method according to  claim 9 , wherein
 the fusing includes:
 concatenating computational weights and biases of the first computation operation and the second computation operation. 
   
     
     
         14 . The method according to  claim 9 , wherein:
 an output of the combined operation leads as input into an activation operation node of the computation graph, the input comprising a dataset having negative data values;   the activation operation node of the computation graph is transformed based on the identifying the input having the dataset having the negative data values, the transforming of the activation operation node includes:
 replacing a first activation function of the activation operation node with a second activation function, wherein:
 (a) the first activation function, if applied at runtime to the dataset having the negative data values, replaces the negative data values with zero, and 
 (b) the second activation function, when applied to the dataset having the negative data values, shifts the negative data values into a positive range of values 
 
   
     
     
         15 . A method for transforming one or more components of a computation network graph, the method comprising:
 identifying a first computation operation node of a computation graph that outputs a dataset having negative data values;   identifying an input leading into a second computation operation node of the computation graph, the input having the dataset having negative data values;   at compile time, transforming the first computation operation node based on identifying the input leading into the second computation operation, the transforming includes:
 (i) installing an activation function within the first computation operation node, the activation function, when applied to the output of the first computation operation node, converts the negative data values of the output of the first computation operation node to positive data values. 
 or 
 (ii) replacing a first activation function of the first computation operation node with a second activation function, the second activation function, when applied to the output of the first computation operation node, converts the negative data values of the output of the first computation operation node to positive data values. 
   
     
     
         16 . The method according to  claim 15 , wherein
 the first computation operation node is implemented by a streaming arithmetic logic unit, and   the activation function comprises a hard sigmoid activation.   
     
     
         17 . The method according to  claim 15 , wherein
 the first computation operation node is implemented by a streaming arithmetic logic unit or a matrix multiply accelerator, and   the first activation function comprises a hardtanh activation and the second activation function replacing the first activation function comprises a hard sigmoid activation.   
     
     
         18 . The method according to  claim 15 , wherein
 the transforming further includes:
 adjusting the weights bias value of the second computation operation node based on an input range of a matrix multiply accelerator implementing the second computation operation node. 
   
     
     
         19 . The method according to  claim 18 , wherein
 the adjusting the weights bias value includes reducing the weights bias value by one-half of the input range of the matrix multiply accelerator.