Systems and methods for accelerating memory transfers and computation efficiency using a computation-informed partitioning of an on-chip data buffer and implementing computation-aware data transfer operations to the on-chip data buffer

Abstract

Systems and methods for implementing accelerated memory transfers in an integrated circuit includes configuring a region of memory of an on-chip data buffer based on a neural network computation graph, wherein configuring the region of memory includes: partitioning the region of memory of the on-chip data buffer to include a first distinct sub-region of memory and a second distinct sub-region of memory; initializing a plurality of distinct memory transfer operations from the off-chip main memory to the on-chip data buffer; executing a first set of memory transfer operations that includes writing a first set of computational components to the first distinct sub-region of memory, and while executing, using the integrated circuit, a leading computation based on the first set of computational components, executing a second set of memory transfer operations to the second distinct sub-region of memory for an impending computation.

What is claimed: 
     
         1 . A method comprising:
 performing, via an integrated circuit, a plurality of memory transfer operations that write computational components of a computational operation to a first region of an on-chip data buffer;   executing, via the integrated circuit, the computational operation based on the computational components stored within the first region of the on-chip data buffer; and   while executing the computational operation, performing a second plurality of memory transfer operations that write computational components of a succeeding computational operation to a second region of the on-chip data buffer.   
     
     
         2 . The method of  claim 1 , wherein:
 when output data resulting from an execution of the computational operation is required for the succeeding computational operation, writing the output data of the computational operation to the second region of the on-chip data buffer.   
     
     
         3 . The method of  claim 1 , wherein:
 the computational operation relates to a first layer within a neural network,   the succeeding computational operation relates to a second layer within the neural network,   an output of the first layer is computed based on one or more computations with a set of weights associated with the first layer and a tensor provided to the first layer,   an output of the second layer depends on one or more computations involving the output of the first layer and a set of weights associated with the second layer, and   performing the plurality of memory transfer operations and the second plurality of memory transfer operations include:
 writing the tensor to a first memory slice of the first region, 
 writing the set of weights associated with the first layer to a second memory slice of the first region, and 
 writing the set of weights associated with the second layer to a first memory slice of the second region. 
   
     
     
         4 . The method of  claim 1 , wherein:
 at least part of the computational components of the succeeding computational operation is stored within the second region of the on-chip data buffer prior to a completion of the computational operation, and   storing the at least part of the computational components prior to the completion of the computational operation minimizes a latency between the completion of the computational operation and a start of an execution of the succeeding computational operation.   
     
     
         5 . The method of  claim 1 , further comprising:
 during an execution of the succeeding computational operation based on the computational components written to the second region of the on-chip data buffer, performing a third plurality of memory transfer operations that write computational components of a new computational operation that is subsequent to the succeeding computational operation to the first region of the on-chip data buffer.   
     
     
         6 . The method of  claim 1 , wherein the plurality of memory transfer operations and the second plurality of memory transfer operations are created based on a configuration of a neural network computation graph. 
     
     
         7 . The method of  claim 1 , wherein:
 the computational components of the computational operation include (i) an input data component and (ii) a distinct sets of coefficients data associated with the computational operation, and   executing the computational operation includes:
 accessing the input data component and the distinct sets of coefficients data from the first region of the on-chip data buffer, and 
 performing one or more computations with the distinct set of coefficients and the input data component accessed from the first region of the on-chip data buffer. 
   
     
     
         8 . The method of  claim 1 , wherein:
 the integrated circuit comprises the on-chip data buffer and a main memory,   the computational components of the computational operation and the computational components of the succeeding computational operation are stored in the main memory, and   performing the plurality and the second plurality of memory transfer operations cause the computational components of the computational operation and the succeeding computational operation to be fetched from the main memory and stored in the on-chip data buffer.   
     
     
         9 . The method of  claim 1 , wherein:
 a neural network computation graph defines a plurality of computational operations including the computational operation and the succeeding computational operation,   each distinct computational operation of the neural network computation graph relates to a distinct layer within a neural network, and   a configuration of the neural network computation graph that identifies graphical edges of flows of input into each distinct computational operation define distinct computational components required by the distinct layer of the neural network.   
     
     
         10 . The method of  claim 1 , wherein:
 the first region of the on-chip data buffer includes a plurality of distinct memory slices, wherein each of the plurality of distinct memory slices relates to a subdivision of the first region of the on-chip data buffer, and   the plurality of memory transfer operations write each of the computational components of the computational operation into a distinct memory slice of the plurality of distinct memory slices.   
     
     
         11 . A method of mitigating computational latency by controlling a flow of data within an integrated circuit, the method comprising:
 partitioning an on-chip data buffer into a plurality of distinct memory regions, wherein:
 a first memory region of the plurality of distinct memory regions is configured to store computational components associated with a target computational operation, and 
 a second memory region of the plurality of distinct memory regions is configured to store computational components associated with a computational operation that is subsequent to the target computational operation; 
   performing a first set of memory transfer operations based on a configuration of a neural network computation graph that writes at least a subset of a plurality of computational components associated with a first computational operation to the first memory region;   executing the first computational operation based at least on the subset of the plurality of computational components stored within the first memory region; and   while executing the first computational operation, performing a second set of memory transfer operations based on the configuration of a neural network computation graph that writes at least a subset of a plurality of computational components associated with a second computational operation to the second memory region, wherein the second computational operation is subsequent to the first computational operation in the neural network computation graph.   
     
     
         12 . The method of claim ii, wherein
 writing the subset of computational components associated with the second computational operation to the second memory region minimizes a latency between a completion of the first computational operation and a start of the second computational operation.   
     
     
         13 . The method of claim ii, wherein:
 the first computational operation relates to a first layer in a neural network and the second computational operation relates to a second layer in the neural network,   a computational output of the second layer depends on one or more computational outputs of the first layer that define input data into the second layer and a set of weights associated with the second layer, and   performing the second set of memory transfer operations include:
 writing the set of weights associated with the second layer to a first memory slice of the second memory region. 
   
     
     
         14 . The method of  claim 13 , wherein the first computational operation computes the output of the first layer, the method further comprising:
 after executing the first computational operation:
 writing the computational output of the first layer to a second memory slice of the second memory region; and 
 executing the second computational operation based on the plurality of computational components associated with the second computational operation, including the set of weights stored in the first memory slice of the second memory region and the computational output of the first layer stored in the second memory slice of the second memory region. 
   
     
     
         15 . A method of reducing latency in an operation of an integrated circuit, the method comprising:
 configuring an on-chip data buffer based on a neural network computation graph, wherein the configuring of the on-chip data buffer includes:
 partitioning the on-chip data buffer to include a first memory region that is configured to receive requisite computational components for a target computation associated with the neural network computation graph; 
 partitioning the on-chip data buffer to include a second memory region that is configured to receive requisite computational components for an impending computation succeeding the target computation in the neural network computation graph; and 
   simultaneously (a) executing the target computation based on reading the first memory region and (b) writing to the second memory region based on the configuration of the on-chip data buffer.   
     
     
         16 . The method of  claim 15 , further comprising:
 writing, into the first memory region, a set of requisite computational components associated with the target computation; and   executing, by at least one processing core of an array of processing cores, the target computation based at least on the set of requisite computational components stored in the first memory region.   
     
     
         17 . The method of  claim 16 , further comprising:
 while executing the target computation:
 writing, to the second memory region, a set of requisite computational components associated with the impending, non-active computation. 
   
     
     
         18 . The method according to  claim 17 , wherein:
 the writing the set of requisite computational components to the second memory region occurs minimizes a latency between the target computation and the impending computation.   
     
     
         19 . The method according to  claim 17 , wherein:
 the set of requisite computational components associated with the target computation includes (i) an input component and (ii) a distinct set of weights associated with a target layer of the neural network computation graph, and   the target computation is a result of one or more operations with the input component and the distinct set of weights.   
     
     
         20 . The method according to  claim 19 , wherein the target computation computes an output data tensor, the method further comprising:
 based on computing the output data tensor, writing the output data tensor to the second memory region of the on-chip data buffer; and   based on writing the output data tensor to the second memory region of the on-chip data buffer, executing, via the at least one processing core of the array of processing cores, the impending, non-active computation based on the output data tensor and the set of requisite computational components associated with the impending, non-active computation.