Electronic device for estimating optical flow and operating method thereof

Abstract

A method of estimating an optical flow includes processing, using an image processing pass, a first image and a second image separately, and estimating the optical flow based on a second image attention feature map of the first image and a second image attention feature map of the second image. The processing using the image processing pass includes extracting a feature map by encoding an image, outputting a first image fusion attention feature map by fusing row relationship information of the image with the image feature map, outputting a first image attention feature map of the image based on the first image fusion attention feature map and the image feature map, outputting a second image fusion attention feature map by fusing column relationship information of the image with the first image attention feature map, and generating a second image attention feature map of the image based on the second image fusion attention feature map and the first image attention feature map.

What is claimed is: 
     
         1 . A method of estimating an optical flow, the method comprising:
 processing, using an image processing pass, a first image and a second image separately; and   estimating the optical flow based on a second image attention feature map of the first image processed using the image processing pass and a second image attention feature map of the second image processed using the image processing pass,   wherein the processing of an image using the image processing pass comprises:
 extracting an image feature map of the image by encoding the image; 
 outputting a first image fusion attention feature map by fusing image width-axial row relationship information of the image with the image feature map; 
 outputting a first image attention feature map of the image based on the first image fusion attention feature map and the image feature map; 
 outputting a second image fusion attention feature map by fusing image height-axial column relationship information of the image with the first image attention feature map; and 
 generating a second image attention feature map of the image based on the second image fusion attention feature map and the first image attention feature map. 
   
     
     
         2 . The method of  claim 1 , wherein the estimating of the optical flow comprises:
 generating a third image fusion attention feature map of the first image by fusing second row relationship information of the second image with the second image attention feature map of the first image based on the second image attention feature map of the first image and an image feature map of the second image;   obtaining a third image attention feature map of the first image based on the third image fusion attention feature map of the first image and the second image attention feature map of the first image;   generating a fourth image fusion attention feature map of the first image by fusing second column relationship information of the second image with the third image attention feature map; and   obtaining a fourth image attention feature map of the first image based on the fourth image fusion attention feature map of the first image and the third image attention feature map of the first image.   
     
     
         3 . The method of  claim 2 , wherein the estimating of the optical flow further comprises:
 generating a third image fusion attention feature map of the second image by fusing first row relationship information of the first image with the second image attention feature map of the second image based on the second image attention feature map of the second image and an image feature map of the first image;   obtaining a third image attention feature map of the second image based on the third image fusion attention feature map of the second image and the second image attention feature map of the second image;   generating a fourth image fusion attention feature map of the second image by fusing first column relationship information of the first image with the third image attention feature map; and   obtaining a fourth image attention feature map of the second image based on the fourth image fusion attention feature map of the second image and the third image attention feature map of the second image.   
     
     
         4 . The method of  claim 3 , wherein the estimating of the optical flow further comprises estimating the optical flow based on the fourth image attention feature map of the first image and the fourth image attention feature map of the second image. 
     
     
         5 . The method of  claim 1 , wherein the processing of the image using the image processing pass further comprises:
 concatenating a row vector for extracting the image width-axial row relationship information with the image feature map of the image;   obtaining a first vector feature map of the image based on the image feature map of the image concatenated with the row vector, wherein the first vector feature map is width-axially separated and attention-learnable;   separating the first vector feature map into a width-axially separated feature map of the image and a first information map corresponding to the image width axial row relationship information; and   generating the first image fusion attention feature map based on the width-axially separated feature map of the image and the first information map.   
     
     
         6 . The method of  claim 1 , wherein the processing of the image using the image processing pass further comprises:
 concatenating a column vector for extracting the image height axial column relationship information with the first image attention feature map of the image;   obtaining a second vector feature map of the image based on the first image attention feature map of the image concatenated with the column vector, wherein the second vector feature map is height-axially separated and attention-learnable;   separating the second vector feature map into a height-axially separated feature map of the image and a second information map corresponding to the image height axial column relationship information; and   generating the second image fusion attention feature map based on the height-axially separated feature map of the image and the second information map.   
     
     
         7 . An apparatus for estimating an optical flow, comprising:
 a memory configured to store one or more instructions; and   a processor communicatively coupled to the memory,   wherein the processor is configured to execute the one or more instructions to:
 process, using an image processing pass, a first image and a second image separately; and 
 estimate the optical flow based on a second image attention feature map of the first image obtained using the image processing pass and a second image attention feature map of the second image obtained using the image processing pass, 
 wherein the image processing pass is configured to:
 extract an image feature map of an image by encoding the image; 
 output a first image fusion attention feature map by fusing image width-axial row relationship information of the image with the image feature map; 
 output a first image attention feature map of the image based on the first image fusion attention feature map and the image feature map; 
 output a second image fusion attention feature map by fusing image height-axial column relationship information of the image with the first image attention feature map; and 
 generate a second image attention feature map of the image based on the second image fusion attention feature map and the first image attention feature map. 
 
   
     
     
         8 . The apparatus of  claim 7 , wherein the processor is further configured to execute the one or more instructions to:
 generate a third image fusion attention feature map of the first image by fusing second row relationship information of the second image with the second image attention feature map of the first image based on the second image attention feature map of the first image and an image feature map of the second image;   obtain a third image attention feature map of the first image based on the third image fusion attention feature map of the first image and the second image attention feature map of the first image;   generate a fourth image fusion attention feature map of the first image by fusing second column relationship information of the second image with the third image attention feature map; and   obtain a fourth image attention feature map of the first image based on the fourth image fusion attention feature map of the first image and the third image attention feature map of the first image.   
     
     
         9 . The apparatus of  claim 8 , wherein the processor is further configured to execute the one or more instructions to:
 generate a third image fusion attention feature map of the second image by fusing first row relationship information of the first image with the second image attention feature map of the second image based on the second image attention feature map of the second image and an image feature map of the first image;   obtain a third image attention feature map of the second image based on the third image fusion attention feature map of the second image and the second image attention feature map of the second image;   generate a fourth image fusion attention feature map of the second image by fusing first column relationship information of the first image with the third image attention feature map; and   obtain a fourth image attention feature map of the second image based on the fourth image fusion attention feature map of the second image and the third image attention feature map of the second image.   
     
     
         10 . The apparatus of  claim 9 , wherein the processor is further configured to execute the one or more instructions to estimate the optical flow based on the fourth image attention feature map of the first image and the fourth image attention feature map of the second image. 
     
     
         11 . The apparatus of  claim 7 , wherein the image processing pass is further configured to:
 concatenate a row vector for extracting the image width-axial row relationship information with the image feature map of the image;   obtain a first vector feature map of the image based on the image feature map of the image concatenated with the row vector, wherein the first vector feature map is width-axially separated and attention-learnable;   separate the first vector feature map into a width-axially separated feature map of the image and a first information map corresponding to the image width axial row relationship information; and   generate the first image fusion attention feature map based on the width-axially separated feature map of the image and the first information map.   
     
     
         12 . The apparatus of  claim 7 , wherein the image processing pass is further configured to:
 concatenate a column vector for extracting the image height axial column relationship information with the first image attention feature map of the image;   obtain a second vector feature map of the image based on the first image attention feature map of the image concatenated with the column vector, wherein the second vector feature map is height-axially separated and attention-learnable;   separate the second vector feature map into a height-axially separated feature map of the image and a second information map corresponding to the image height axial column relationship information; and   generate the second image fusion attention feature map based on the height-axially separated feature map of the image and the second information map.   
     
     
         13 . A method of estimating an optical flow, the method comprising:
 generating a plurality of first clusters comprising features of a first image feature map representing a first image by clustering the first image feature map;   determining a pixel in a second image corresponding to a center feature from among the features respectively included in the plurality of first clusters from a second image feature map representing the second image;   obtaining a loss matrix based on the plurality of first clusters and the determined pixel in the second image; and   estimating the optical flow of the first image and the second image based on the loss matrix.   
     
     
         14 . The method of  claim 13 , wherein the generating of the plurality of first clusters comprises:
 dividing the features of the first image feature map of the first image into a plurality of subsets; and   dividing the features of the first image feature map of the first image into the plurality of first clusters.   
     
     
         15 . The method of  claim 13 , wherein the determining of the pixel in the second image comprises:
 determining one or more features most associated with the center feature of each cluster among the plurality of first clusters in an image feature matrix of the second image.   
     
     
         16 . The method of  claim 13 , wherein the obtaining of the loss matrix comprises calculating a correlation between the center feature and the pixel in the second image. 
     
     
         17 . An apparatus for estimating an optical flow, comprising:
 a memory configured to store one or more instructions; and   a processor communicatively coupled to the memory,   wherein the processor is configured to execute the one or more instructions to:
 generate a plurality of first clusters comprising features of a first feature map representing a first image by clustering the first feature map; 
 determine a pixel in a second image corresponding to a center feature from among the features respectively included in the plurality of first clusters from a second feature map representing the second image; 
 obtain a loss matrix based on the plurality of first clusters and the determined pixel in the second image; and 
 estimate the optical flow of the first image and the second image based on the loss matrix. 
   
     
     
         18 . The apparatus of  claim 17 , wherein the processor is further configured to execute the one or more instructions to:
 divide the features of the first feature map of the first image into a plurality of subsets; and   divide the features of the first feature map of the first image into the plurality of first clusters.   
     
     
         19 . The apparatus of  claim 17 , wherein the processor is further configured to execute the one or more instructions to:
 determine one or more features most associated with the center feature of each cluster among the plurality of first clusters in an image feature matrix of the second image.   
     
     
         20 . The apparatus of  claim 17 , wherein the processor is further configured to execute the one or more instructions to:
 calculate a correlation between the center feature and the pixel in the second image.