Apparatus and method for generating super resolution image using orientation adaptive parallel neural networks

Abstract

A method for generating a super resolution image may comprise up-scaling an input low resolution image; determining a directivity for each patch included in the up-scaled image; selecting an orientation-specified neural network or an orientation-non-specified neural network according to the directivity of the patch; applying the selected neural network to the patch; and obtaining a super resolution image by combining one or more patches output from the orientation-specified neural network and the orientation-non-specified neural network.

What is claimed is: 
     
       1. A method for generating a super resolution image, the method comprising:
 up-scaling an input low resolution image; 
 determining a directivity for each patch included in the up-scaled image; 
 selecting an orientation-specified neural network or an orientation-non-specified neural network according to the directivity of each patch; 
 applying the selected neural network to each patch; and 
 obtaining a super resolution image by combining one or more of each patch output from the orientation-specified neural network or the orientation-non-specified neural network based on the selecting step. 
 
     
     
       2. The method according to  claim 1 , wherein the applying of the selected neural network to each patch comprises applying the orientation-specified neural network to each patch having a specific directivity. 
     
     
       3. The method according to  claim 2 , wherein the applying of the orientation-specified neural network to each patch having a specific directivity comprises:
 rotating each patch so that an orientation of each patch becomes a preconfigured orientation learned by the orientation-specified neural network; 
 applying iterative architectures to each rotated patch; 
 applying a fully-connected layer to a feature map output from the iterative architectures so that a size and a shape of the feature map become identical to a size and a shape of the up-scaled image; and 
 re-converting each patch to an original orientation. 
 
     
     
       4. The method according to  claim 3 , wherein the applying of the orientation-specified neural network to the patch having a specific directivity further comprises:
 inserting an outline to the patch before rotating each patch; and 
 removing the outline from each patch whose angle has been reconverted to the original orientation. 
 
     
     
       5. The method according to  claim 1 , wherein the orientation-specified neural network includes neural network parameters learned using high directivity patches having a preconfigured directivity among a plurality of patches in a training image converted using bicubic interpolation. 
     
     
       6. The method according to  claim 1 , wherein the orientation-non-specified neural network includes neural network parameters learned using low directivity patches among a plurality of patches in a training image converted using bicubic interpolation. 
     
     
       7. The method according to  claim 1 , wherein the determining of the directivity for each patch included in the up-scaled image comprises:
 calculating a size and an orientation of a gradient for each pixel in each patch; 
 deriving a histogram by calculating a frequency for a gradient orientation for pixels having a calculated gradient size equal to or greater than a preconfigured size; and 
 determining the directivity of the patch as a high directivity or a low directivity according to whether a ratio of a first maximum value and a second maximum value of the frequency in the histogram is greater than or equal to a preconfigured ratio. 
 
     
     
       8. The method according to  claim 7 , wherein the selecting of the orientation-specified neural network or the orientation-non-specified neural network according to the directivity of the patch comprises:
 selecting the orientation-specified neural network for each patch having the high directivity; and 
 selecting the orientation-non-specified neural network for each patch having the low directivity. 
 
     
     
       9. The method according to  claim 1 , wherein the applying of the selected neural network to each patch comprises applying the orientation-non-specified neural network to each patch not having a specific directivity. 
     
     
       10. The method according to  claim 9 , wherein the applying of the orientation-non-specified neural network to each patch not having a specific directivity comprises:
 applying iterative architectures to an input patch; and 
 applying a fully-connected layer to a feature map output from the iterative architectures. 
 
     
     
       11. The method according to  claim 3 , wherein the iterative architectures include at least one layer architecture, and the layer architecture includes a convolution, a batch normalization (BN), and a rectified linear unit (ReLU), and generates a feature map of the input patch. 
     
     
       12. A super resolution image generation apparatus, the apparatus comprising:
 a processor; and
 a memory storing at least one instruction executable by the processor, 
 wherein when executed by the processor, the at least one instruction causes the processor to: 
 
 up-scale an input low resolution image; 
 determine a directivity for each patch included in the up-scaled image; 
 select an orientation-specified neural network or an orientation-non-specified neural network according to the directivity of each patch; 
 apply the selected neural network to each patch; and 
 obtain a super resolution image by combining one or more of each patch output from the orientation-specified neural network or the orientation-non-specified neural network based on the selecting step. 
 
     
     
       13. The super resolution image generation apparatus according to  claim 12 , wherein in the applying of the selected neural network to each patch, the at least one instruction further causes the processor to apply the orientation-specified neural network to each patch having a specific directivity. 
     
     
       14. The super resolution image generation apparatus according to  claim 13 , wherein in the applying of the orientation-specified neural network to each patch having a specific directivity, the at least one instruction further causes the processor to:
 rotate each patch so that an orientation of the patch becomes a preconfigured orientation learned by the orientation-specified neural network; 
 apply iterative architectures to each rotated patch; 
 apply a fully-connected layer to a feature map output from the iterative architectures so that a size and a shape of the feature map become identical to a size and a shape of the up-scaled image; and 
 re-convert each patch to an original orientation. 
 
     
     
       15. The super resolution image generation apparatus according to  claim 14 , wherein in the applying of the orientation-specified neural network to each patch having a specific directivity, the at least one instruction further causes the processor to:
 insert an outline to each patch before rotating the patch; and 
 remove the outline from each patch whose angle has been reconverted to the original orientation. 
 
     
     
       16. The super resolution image generation apparatus according to  claim 12 , wherein the orientation-specified neural network includes neural network parameters learned using high directivity patches having a preconfigured directivity among a plurality of patches in a training image converted using bicubic interpolation. 
     
     
       17. The super resolution image generation apparatus according to  claim 12 , wherein the orientation-non-specified neural network includes neural network parameters learned using low directivity patches among a plurality of patches in a training image converted using bicubic interpolation. 
     
     
       18. The super resolution image generation apparatus according to  claim 12 , wherein in the determining of the directivity for each patch included in the up-scaled image, the at least one instruction further causes the processor to:
 calculate a size and an orientation of a gradient for each pixel in each patch; 
 derive a histogram by calculating a frequency for a gradient orientation for pixels having a calculated gradient size equal to or greater than a preconfigured size; and 
 determine the directivity of each patch as a high directivity or a low directivity according to whether a ratio of a first maximum value and a second maximum value of the frequency in the histogram is greater than or equal to a preconfigured ratio. 
 
     
     
       19. The super resolution image generation apparatus according to  claim 12 , wherein in the applying of the selected neural network to each patch, the at least one instruction further causes the processor to apply the orientation-non-specified neural network to each patch not having a specific directivity. 
     
     
       20. The super resolution image generation apparatus according to  claim 19 , wherein in the applying of the orientation-non-specified neural network to each patch not having a specific directivity, the at least one instruction further causes the processor to apply iterative architectures to an input patch; and apply a fully-connected layer to a feature map output from the iterative architectures.