Device and method with neural network compilation

Abstract

A device and method with neural network compilation is provided. The method includes detecting for a skip connection in a neural network, where the skip connection is configured to output first data to a second layer in the neural network, wherein the first data is output from a first layer in the neural network, and wherein the second layer is beyond a hierarchically subsequent layer of the first layer in the neural network, determining, in the neural network, a decompression layer set including a layer, which is configured in the neural network to be input second data with a dimension smaller a dimension of the first data, that is among one or more layers hierarchically preceding the first layer in the neural network, and generating a compiled version of the neural network by changing the skip connection dependent on the determined decompression layer set.

What is claimed is: 
     
         1 . A processor-implemented method, the method comprising:
 detecting for a skip connection in a neural network, where the skip connection is configured to output first data to a second layer in the neural network, wherein the first data is output from a first layer in the neural network, and wherein the second layer is beyond a hierarchically subsequent layer of the first layer in the neural network;   determining, in the neural network, a decompression layer set comprising a layer, which is configured in the neural network to be input second data with a dimension smaller a dimension of the first data, that is among one or more layers hierarchically preceding the first layer in the neural network; and   generating a compiled version of the neural network by changing the skip connection dependent on the determined decompression layer set.   
     
     
         2 . The method of  claim 1 , wherein the determining of the decompression layer set comprises:
 detecting, among the one or more layers, the layer as a decompression layer that is configured to be input data having a dimension smaller than the dimension of the first data by more than a threshold dimension; and   determining all layers, sequentially from the decompression layer to the first layer in the neural network, as the decompression layer set.   
     
     
         3 . The method of  claim 1 , wherein the changing of the skip connection comprises:
 changing the detected skip connection, from the output of the first data from the first layer to the second layer, into a different skip connection using a copied layer set, which is a copy of the determined decompression layer set, that is configured to be input the second data and provide third data output from the copied layer set to the second layer.   
     
     
         4 . The method of  claim 1 , further comprising:
 detecting, in the neural network, for an activation layer set comprising a decompression layer that is configured to decompresses the second data, an activation layer that is configured to apply an activation function to an output of the decompression layer, and a compression layer that is configured to compress an output of the activation layer in the neural network,   wherein the generating of the compiled version of the neural network further comprises changing the detected activation layer set into a fusion layer that is configured in the compiled version of the neural network to perform decompression, application of an activation function, and compression for each of tiles extracted from the second data.   
     
     
         5 . The method of  claim 4 ,
 wherein the detecting for the activation layer set comprises detecting for a concatenation layer set comprising a first decompression layer, a first activation layer that is configured to apply a first activation function to an output of the first decompression layer, a second decompression layer, a second activation layer that configured to apply a second activation function to an output of the second decompression layer, a first concatenation layer that is configured to concatenate an output of the first activation layer and an output of the second activation layer, and a compression layer that is configured to compress an output of the first concatenation layer in the neural network, and   wherein:
 the changing of the detected activation layer set comprises changing the detected concatenation layer set into the fusion layer, with the first decompression layer being the decompression layer, the first activation layer being the activation layer, and the first compression layer is the compression layer; or 
 wherein the generating of the compiled version of the neural network further comprises changing the detected concatenation layer set into another fusion layer that is configured to performs a corresponding decompression, application of a corresponding activation function, and corresponding compression for each of tiles extracted from data that was configured in the neural network to be input to the concatenation layer set. 
   
     
     
         6 . The method of  claim 5 ,
 wherein, in response to a determination that the second layer is configured to concatenate an output of the first activation layer and an output of the second activation layer, detecting the second layer as the first concatenation layer and performing the changing of the detected concatenation layer set into the fusion layer, where the performing of the changing of the detected concatenation layer set into the fusion layer occurs after the changing of the skip connection,   wherein changed skip connection is configured to input the second data to the second decompression layer, and output a result of the second activation layer to the second layer, and   wherein the detecting for the concatenation layer set comprises:
 detecting for the first decompression layer and the first activation layer between the first layer and the second layer in the neural network; and 
   detecting for the second decompression layer and the second activation layer in the changed skip connection.   
     
     
         7 . The method of  claim 1 , further comprising:
 detecting for a concatenation layer set comprising a first decompression layer, a first activation layer that is configured to apply a first activation function to an output of the first decompression layer, a second decompression layer, a second activation layer that configured to apply a second activation function to an output of the second decompression layer, a first concatenation layer that is configured to concatenate an output of the first activation layer and an output of the second activation layer, and a compression layer that is configured to compress an output of the first concatenation layer in the neural network, where the detection of the second decompression layer and the detection of the second activation layers are performed after performance of the changing the skip connection; and   in response to the first activation function being determined to be same as the second activation function, replacing the concatenation layer set with a second concatenation layer that is configured to concatenate an input of the first decompression layer and an input of the second decompression layer, and a fusion layer that is configured to perform decompression, application of the first activation function, and compression for each of tiles extracted from an output of the second concatenation layer.   
     
     
         8 . The method of  claim 1 , further comprising:
 detecting for a concatenation layer set comprising a first decompression layer, a first activation layer that is configured to apply a first activation function to an output of the first decompression layer, a second decompression layer, a second activation layer that configured to apply a second activation function to an output of the second decompression layer, a first concatenation layer that is configured to concatenate an output of the first activation layer and an output of the second activation layer, and a compression layer that is configured to compress an output of the first concatenation layer in the neural network, where the detection of the second decompression layer and the detection of the second activation layers are performed after performance of the changing the skip connection; and   in response to a determination that the first activation function is different from the second activation function, replacing the concatenation layer set with a first fusion layer that is configured to perform decompression, application of the first activation function, and compression for each of tiles for an input of the first decompression layer, a second fusion layer that is configured to perform decompression, application of the second activation function, and compression for each of tiles for an input of the second decompression layer, and a summation layer that is configured to sum an output of the first fusion layer and an output of the second fusion layer.   
     
     
         9 . The method of  claim 1 , wherein the detecting for the skip connection comprises, in response to a determination that the neural network is configured to use the first data in the second layer after the first data is output by the first layer, detecting for the skip connection between the first layer and the second layer based on a determined hierarchical distance between the first layer and the second layer. 
     
     
         10 . The method of  claim 1 , wherein, with the neural network comprising the skip connection, the generating of the compiled version of the neural network further comprises determining whether to change the skip connection based on a result of a comparing of a computational load of a copied layer set, which is a copy of the determined decompression layer set, with a threshold computation overhead. 
     
     
         11 . The method of  claim 1 , wherein, with the neural network comprising the skip connection, the generating of the compiled version of the neural network further comprises determining whether to change the skip connection based on a result of a comparing of memory usage of a copied layer set, which is a copy of the determined decompression layer set, with a threshold memory overhead. 
     
     
         12 . A non-transitory computer-readable storage medium storing code that, in response to being executed by one or more processors, cause the one or more processors to perform the method of  claim 1 . 
     
     
         13 . A processor-implemented method, the method comprising:
 detecting for an activation layer set in a neural network, the activation layer set comprising a decompression layer that is configured to decompresses input data, an activation layer that is configured to apply an activation function to an output of the decompression layer, and a compression layer that is configured to compress data that is based on an output of the activation layer; and   generating a compiled version of the neural network by changing the activation layer set into a fusion layer that is configured to perform decompression, application of an activation function, and compression for each of tiles that are dependent of the input data.   
     
     
         14 . The method of  claim 13 ,
 wherein the detecting for the activation layer set comprises detecting for the decompression layer, the activation layer, a pooling layer that that is configured to apply a pooling to the output of the activation layer, and the compression layer that is configured to compresses an output of the pooling layer, and   wherein the fusion layer is further configured to perform an application of pooling.   
     
     
         15 . The method of  claim 13 , wherein
 the detecting for the activation layer set comprises detecting for a concatenation layer set comprising a first decompression layer, a first activation layer that is configured to apply a first activation function to an output of the first decompression layer, a second decompression layer, a second activation layer that is configured to apply a second activation function to an output of the second decompression layer, a first concatenation layer that is configured to concatenate an output of the first activation layer and an output of the second activation layer, and the compression layer that is configured to compress an output of the first concatenation layer in the neural network, and   wherein the changing of the activation layer set comprises changing the concatenation layer set into the fusion layer.   
     
     
         16 . The method of  claim 13 ,
 wherein the detecting for the activation layer set further comprises detecting for a concatenation layer set comprising a first decompression layer, a first activation layer that is configured to apply a first activation function to an output of the first decompression layer, a second decompression layer, a second activation layer that configured to apply a second activation function to an output of the second decompression layer, a first concatenation layer that is configured to concatenate an output of the first activation layer and an output of the second activation layer, and the compression layer that is configured to compress data that is based on an output of the first concatenation layer in the neural network, and   wherein the changing of the activation layer set comprises, in response to a determination that the first activation function is same as the second activation function, replacing the concatenation layer set with a second concatenation layer that is configured to concatenate an input of the first decompression layer and an input of the second decompression layer, and the fusion layer that is configured to perform an application of the first activation function as the application of the activation function, and the compression for each of the tiles that are extracted from an output of the second concatenation layer.   
     
     
         17 . The method of  claim 13 ,
 wherein the detecting for the activation layer set further comprises detecting for a concatenation layer set comprising a first decompression layer, a first activation layer that is configured to apply a first activation function to an output of the first decompression layer, a second decompression layer, a second activation layer that configured to apply a second activation function to an output of the second decompression layer, a first concatenation layer that is configured to concatenate an output of the first activation layer and an output of the second activation layer, and the compression layer that is configured to compress data that is based on an output of the first concatenation layer in the neural network, and   wherein the changing of the activation layer set comprises, in response to a determination that the first activation function is different from the second activation function, replacing the concatenation layer set with a first fusion layer that is configured to perform decompression, application of the first activation function, and compression for each of tiles for an input of the first decompression layer, a second fusion layer that is configured to perform decompression, application of the second activation function, and compression for each of tiles for an input of the second decompression layer, and a summation layer that is configured to sum an output of the first fusion layer and an output of the second fusion layer.   
     
     
         18 . The method of  claim 13 , wherein the generating of the compiled version of the neural network further comprises changing a detected skip connection, based on a determined decompression layer set, before the changing of the activation layer set. 
     
     
         19 . An electronic device comprising:
 one or more processors; and   a memory storing code,   wherein the code is configured to, in response to being executed by the one or more processors, cause the one or more processors to:
 detect for a skip connection in a neural network, where the skip connection is configured to output first data to a second layer in the neural network, wherein the first data is output from a first layer in the neural network, and wherein the second layer is beyond a hierarchically subsequent layer of the first layer in the neural network; 
 determine, in the neural network, a decompression layer set comprising a layer, which is configured in the neural network to be input second data with a dimension smaller a dimension of the first data, that is among one or more layers hierarchically preceding the first layer in the neural network; and 
 generate a compiled version of the neural network by changing the skip connection dependent on the determined decompression layer set. 
   
     
     
         20 . The electronic device of  claim 19 , wherein, for the changing of the skip connection, the instructions are configured to, in response to being executed by the processor, cause the one or more processors to:
 change the detected skip connection, from the output of the first data from the first layer to the second layer, into a different skip connection using a copied layer set, which is a copy of the determined decompression layer set, that is configured to be input the second data and provide third data output from the copied layer set to the second layer.   
     
     
         21 . The electronic device of  claim 19 , wherein the code is configured to, in response to being executed by the one or more processors, cause the one or more processors to:
 detect, in the neural network, for an activation layer set comprising a decompression layer that is configured to decompresses the second data, an activation layer that is configured to apply an activation function to an output of the decompression layer, and a compression layer that is configured to compresses an output of the activation layer in the neural network,   wherein the generation of the compiled version of the neural network further comprises a changing of the detected activation layer set into a fusion layer that is configured in the compiled version of the neural network to perform decompression, application of an activation function, and compression for each of tiles extracted from the second data.