Ai encoding apparatus and operation method of the same, and ai decoding apparatus and operation method of the same

Abstract

Provided is an artificial intelligence (AI) decoding apparatus including a memory storing one or more instructions; and a processor configured to execute the one or more instructions to, when an image is input to a second DNN including a plurality of layers, obtain first result values based on an operation between the image and a first filter kernel and obtain second result values based on an operation between the image and a second filter kernel, from a first layer including the first and second filter kernels from among the plurality of layers, perform normalization by transforming the first result values into first values by using a first scale factor, and, perform normalization by transforming the second result values into second values by using a second scale factor, transform the first values and the second values into integer values included in a preset range.

What is claimed is: 
     
         1 . An artificial intelligence (AI) decoding apparatus comprising:
 a memory storing one or more instructions; and   a processor configured to execute the stored one or more instructions to:
 obtain image data a first image that is obtained by downscaling an original image by using a first deep neural network (DNN), 
 obtain a second image as a reconstructed image of the first image, based on the image data, 
 input the second image to a second DNN comprising a plurality of layers comprising a first layer and a second layer, the first layer comprising a first filter kernel and a second filter kernel, 
 obtain first result values based on an operation between the second image and the first filter kernel, and obtain second result values based on an operation between the second image and the second filter kernel, 
 perform normalization by transforming the first result values into first values by using a first scale factor, and perform normalization by transforming the second result values into second values by using a second scale factor, 
 transform the first values and the second values into integer values included in a preset range, and 
 input the integer values to the second layer, obtain third result values based on the integer values in the second layer, and obtain a third image by upscaling the second image, based on the third result values, 
   wherein the first scale factor and the second scale factor are set in association with scale factors of the first DNN.   
     
     
         2 . The AI decoding apparatus of  claim 1 , wherein
 the second DNN is a network trained in connection with the first DNN, based on an image obtained during training of the first DNN, and   the first scale factor and the second scale factor are set via training of the second DNN and the first DNN.   
     
     
         3 . The AI decoding apparatus of  claim 1 , wherein the first scale factor and the second scale factor are trained such that a normal distribution of the first values becomes equal to a normal distribution of the second values. 
     
     
         4 . The AI decoding apparatus of  claim 1 , wherein the processor is further configured to execute the one or more instructions to clip the first values and the second values to the preset range and transform the clipped first and second values into the integer values. 
     
     
         5 . The AI decoding apparatus of  claim 4 , wherein the processor is further configured to execute the one or more instructions to transform the clipped first and second values into the integer values by rounding off the clipped first and second values. 
     
     
         6 . The AI decoding apparatus of  claim 1 , wherein the processor is further configured to execute the one or more instructions to store the integer values in the memory. 
     
     
         7 . The AI decoding apparatus of  claim 6 , wherein a size of the memory that stores the integer values is determined based on the preset range. 
     
     
         8 . The AI decoding apparatus of  claim 1 , wherein the processor is further configured to execute the one or more instructions to:
 generate a first feature map by performing a convolution operation between the second image and the first filter kernel, and obtain the first result values by applying a first activation function to the first feature map, and   generate a second feature map by performing a convolution operation between the second image and the second filter kernel, and obtain the second result values by applying a second activation function to the second feature map.   
     
     
         9 . The AI decoding apparatus of  claim 8 , wherein
 the first activation function and the second activation function are expressed as   
       
         
           
             
               
                 f 
                  
                 
                   ( 
                   x 
                   ) 
                 
               
               = 
               
                 { 
                 
                   
                     
                       
                         
                           ax 
                           , 
                           
                             x 
                             < 
                             0 
                           
                         
                       
                     
                     
                       
                         
                           bx 
                           , 
                           
                             x 
                             ≥ 
                             0 
                           
                         
                       
                     
                   
                   , 
                 
               
             
           
         
         wherein f(x) denotes the first activation function or the second activation function, a and b denote slopes that have different values, and 
         wherein a value of b is determined via training of the second DNN. 
       
     
     
         10 . The AI decoding apparatus of  claim 9 , wherein the value of b in the first activation function is different from the value of b in the second activation function. 
     
     
         11 . An artificial intelligence (AI) encoding apparatus comprising:
 a memory storing one or more instructions; and   a processor configured to execute the stored one or more instructions to:
 input an original image to a first deep neural network (DNN) comprising a plurality of layers comprising a first layer and a second layer, the first layer comprising a first filter kernel and a second filter kernel, 
 obtain first result values based on an operation between the original image and the first filter kernel, and obtain second result values based on an operation between the original image and the second filter kernel, 
 perform normalization by transforming the first result values into first values by using a first scale factor, and perform normalization by transforming the second result values into second values by using a second scale factor, 
 transform the first values and the second values into integer values included in a preset range, 
 input the integer values to the second layer, obtain third result values based on the integer values in the second layer, and obtain a first image by downscaling the original image, based on the third result values, and 
 encode the first image, 
   wherein the first scale factor and the second scale factor are set in association with scale factors of a second DNN that is configured to upscale the first image.   
     
     
         12 . The AI encoding apparatus of  claim 11 , wherein
 the first DNN is a network trained in connection with the second DNN, based on loss information that is obtained during training of the second DNN, and   the first scale factor and the second scale factor are set via training of the second DNN and the first DNN.   
     
     
         13 . The AI encoding apparatus of  claim 12 , wherein the first DNN is a network trained based on first loss information generated by AI downscaling during training of the first DNN and second loss information generated by AI upscaling during training of the second DNN. 
     
     
         14 . The AI encoding apparatus of  claim 11 , wherein the first scale factor and the second scale factor are trained such that a normal distribution of the first values becomes equal to a normal distribution of the second values. 
     
     
         15 . The AI encoding apparatus of  claim 11 , wherein the processor is further configured to execute the one or more instructions to clip the first values and the second values to the preset range and transform the clipped first and second values into the integer values. 
     
     
         16 . The AI encoding apparatus of  claim 15 , wherein the processor is further configured to execute the one or more instructions to transform the clipped first and second values into the integer values by rounding off the clipped first and second values. 
     
     
         17 . The AI encoding apparatus of  claim 11 , wherein the processor is further configured to execute the one or more instructions to store the integer values in the memory. 
     
     
         18 . The AI encoding apparatus of  claim 17 , wherein a size of the memory that stores the integer values is determined based on the preset range. 
     
     
         19 . An operation method of an artificial intelligence (AI) decoding apparatus, the operation method comprising:
 obtaining image data corresponding to a first image that is obtained by downscaling an original image by using a first deep neural network (DNN);   obtaining a second image as a reconstructed image of the first image, based on the image data;   inputting the second image to a second DNN comprising a plurality of layers comprising a first layer and a second layer, the first layer comprising a first filter kernel and a second filter kernel;   obtaining first result values based on an operation between the second image and the first filter kernel, and obtaining second result values based on an operation between the second image and the second filter kernel;   performing normalization by transforming the first result values into first values by using a first scale factor, and performing normalization by transforming the second result values into second values by using a second scale factor;   transforming the first values and the second values into integer values included in a preset range;   inputting the integer values to the second layer, and obtaining third result values based on the integer values in the second layer; and   obtaining a third image by upscaling the second image, based on the third result values,   wherein the first scale factor and the second scale factor are set in association with scale factors of the first DNN.   
     
     
         20 . An operation method of an artificial intelligence (AI) encoding apparatus, the operation method comprising:
 inputting an original image to a first deep neural network (DNN) comprising a plurality of layers comprising a first layer and a second layer, the first layer comprising a first filter kernel and a second filter kernel;   obtaining first result values based on an operation between the original image and the first filter kernel and obtaining second result values based on an operation between the original image and the second filter kernel;   performing normalization by transforming the first result values into first values by using a first scale factor, and performing normalization by transforming the second result values into second values by using a second scale factor;   transforming the first values and the second values into integer values included in a preset range;   inputting the integer values to the second layer, and obtaining third result values based on the integer values in the second layer;   obtaining a first image by downscaling the original image, based on the third result values; and   encoding the first image,   wherein the first scale factor and the second scale factor are set in association with scale factors of the second DNN.