Compiler techniques for mapping program code to a high performance, power efficient, programmable image processing hardware platform

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for restructuring an image processing pipeline. The method includes compiling program code targeted for an image processor having programmable stencil processors composed of respective two-dimensional execution lane and shift register circuit structures. The program code is to implement a directed acyclic graph and is composed of multiple kernels that are to execute on respective ones of the stencil processors, wherein the compiling includes performing any of: horizontal fusion of kernels; vertical fusion of kernels; fission of one of the kernels into multiple kernels; spatial partitioning of a kernel into multiple spatially partitioned kernels; or splitting the directed acyclic graph into smaller graphs.

1 - 20 . (canceled) 
     
     
         21 . A method performed by one or more computers, the method comprising:
 receiving instructions that define an original processing pipeline to be executed by a device having a plurality of processors, the original processing pipeline comprising a plurality of kernels to be executed in a particular order;   determining that the original processing pipeline satisfies one or more graph-splitting criteria;   in response, generating multiple processing pipelines including:
 generating a first modified processing pipeline including modifying one or more store instructions of a first kernel in the original processing pipeline that reference an internal line buffer that is internal to the device to be one or more respective store instructions that reference memory external to the device; and 
 generating a second modified processing pipeline including modifying one or more load instructions of a second kernel in the original processing pipeline that reference the internal line buffer to be load instructions that reference the memory external to the device. 
   
     
     
         22 . The method of  claim 21 , wherein modifying the one or more load instructions comprises modifying the one or more load instructions to be instructions executed by the device after the first kernel has been executed. 
     
     
         23 . The method of  claim 21 , wherein determining that the original processing pipeline satisfies one or more graph-splitting criteria comprises determining that an amount of data generated by the original processing pipeline exceeds an internal memory size of the device. 
     
     
         24 . The method of  claim 23 , wherein determining that the amount of data generated by the original processing pipeline exceeds the internal memory size comprises determining that the amount of data generated exceeds a memory size of a line buffer of the device, a memory size of a sheet generator of the device, or an internal memory size of one of the plurality of processors of the device. 
     
     
         25 . The method of  claim 21 , wherein determining that the original processing pipeline satisfies one or more graph-splitting criteria comprises determining that a kernel of the plurality of kernels exceeds a predetermined measure of computational complexity. 
     
     
         26 . The method of  claim 21 , further comprising assigning a same processor of the device to execute a first kernel belonging to the first modified processing pipeline and to subsequently execute a second kernel belonging to the second modified processing pipeline. 
     
     
         27 . The method of  claim 21 , wherein each of the plurality of processors is interconnected to a line buffer unit, and wherein modifying the one or more load instructions comprises modifying the one or more load instructions to be instructions for loading data from memory external to the device into the line buffer unit or loading the data from the line buffer unit into the memory external to the device. 
     
     
         28 . The method of  claim 21 , wherein each of the plurality of processors is interconnected to a line buffer unit, and wherein modifying the one or more store instructions comprises modifying the one or more store instructions to be instructions to store data representing output from the first kernel in the line buffer unit. 
     
     
         29 . A system comprising:
 one or more computers and one or more storage devices on which are stored instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:   receiving instructions that define an original processing pipeline to be executed by a device having a plurality of processors, the original processing pipeline comprising a plurality of kernels to be executed in a particular order;   determining that the original processing pipeline satisfies one or more graph-splitting criteria;   in response, generating multiple processing pipelines including:
 generating a first modified processing pipeline including modifying one or more store instructions of a first kernel in the original processing pipeline that reference an internal line buffer that is internal to the device to be one or more respective store instructions that reference memory external to the device; and 
 generating a second modified processing pipeline including modifying one or more load instructions of a second kernel in the original processing pipeline that reference the internal line buffer to be load instructions that reference the memory external to the device. 
   
     
     
         30 . The system of  claim 29 , wherein modifying the one or more load instructions comprises modifying the one or more load instructions to be instructions executed by the device after the first kernel has been executed. 
     
     
         31 . The system of  claim 29 , wherein determining that the original processing pipeline satisfies one or more graph-splitting criteria comprises determining that an amount of data generated by the original processing pipeline exceeds an internal memory size of the device. 
     
     
         32 . The system of  claim 31 , wherein determining that the amount of data generated by the original processing pipeline exceeds the internal memory size comprises determining that the amount of data generated exceeds a memory size of a line buffer of the device, a memory size of a sheet generator of the device, or an internal memory size of one of the plurality of processors of the device. 
     
     
         33 . The system of  claim 29 , wherein determining that the original processing pipeline satisfies one or more graph-splitting criteria comprises determining that a kernel of the plurality of kernels exceeds a predetermined measure of computational complexity. 
     
     
         34 . The system of  claim 29 , wherein the operations further comprise assigning a same processor of the device to execute a first kernel belonging to the first modified processing pipeline and to subsequently execute a second kernel belonging to the second modified processing pipeline. 
     
     
         35 . The system of  claim 29 , wherein each of the plurality of processors is interconnected to a line buffer unit, and wherein modifying the one or more load instructions comprises modifying the one or more load instructions to be instructions for loading data from memory external to the device into the line buffer unit or loading the data from the line buffer unit into the memory external to the device. 
     
     
         36 . The system of  claim 29 , wherein each of the plurality of processors is interconnected to a line buffer unit, and wherein modifying the one or more store instructions comprises modifying the one or more store instructions to be instructions to store data representing output from the first kernel in the line buffer unit. 
     
     
         37 . One or more non-transitory computer-readable storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:
 receiving instructions that define an original processing pipeline to be executed by a device having a plurality of processors, the original processing pipeline comprising a plurality of kernels to be executed in a particular order;   determining that the original processing pipeline satisfies one or more graph-splitting criteria;   in response, generating multiple processing pipelines including:
 generating a first modified processing pipeline including modifying one or more store instructions of a first kernel in the original processing pipeline that reference an internal line buffer that is internal to the device to be one or more respective store instructions that reference memory external to the device; and 
 generating a second modified processing pipeline including modifying one or more load instructions of a second kernel in the original processing pipeline that reference the internal line buffer to be load instructions that reference the memory external to the device. 
   
     
     
         38 . The computer-readable storage media of  claim 37 , wherein modifying the one or more load instructions comprises modifying the one or more load instructions to be instructions executed by the device after the first kernel has been executed. 
     
     
         39 . The computer-readable storage media of  claim 37 , wherein determining that the original processing pipeline satisfies one or more graph-splitting criteria comprises determining that an amount of data generated by the original processing pipeline exceeds an internal memory size of the device. 
     
     
         40 . The computer-readable storage media of  claim 39 , wherein determining that the amount of data generated by the original processing pipeline exceeds the internal memory size comprises determining that the amount of data generated exceeds a memory size of a line buffer of the device, a memory size of a sheet generator of the device, or an internal memory size of one of the plurality of processors of the device. 
     
     
         41 . The computer-readable storage media of  claim 37 , wherein determining that the original processing pipeline satisfies one or more graph-splitting criteria comprises determining that a kernel of the plurality of kernels exceeds a predetermined measure of computational complexity. 
     
     
         42 . The computer-readable storage media of  claim 37 , wherein the operations further comprise assigning a same processor of the device to execute a first kernel belonging to the first modified processing pipeline and to subsequently execute a second kernel belonging to the second modified processing pipeline. 
     
     
         43 . The computer-readable storage media of  claim 37 , wherein each of the plurality of processors is interconnected to a line buffer unit, and wherein modifying the one or more load instructions comprises modifying the one or more load instructions to be instructions for loading data from memory external to the device into the line buffer unit or loading the data from the line buffer unit into the memory external to the device. 
     
     
         44 . The computer-readable storage media of  claim 37 , wherein each of the plurality of processors is interconnected to a line buffer unit, and wherein modifying the one or more store instructions comprises modifying the one or more store instructions to be instructions to store data representing output from the first kernel in the line buffer unit.