Viewpoint image processing method and related device

Abstract

A viewpoint image processing method and a related device are provided, and relate to the artificial intelligence/computer vision field. The method includes: obtaining a preset quantity of first viewpoint images; obtaining a geometric feature matrix between the preset quantity of first viewpoint images; generating an adaptive convolution kernel corresponding to each pixel of the preset quantity of first viewpoint images based on the geometric feature matrix and location information of a to-be-synthesized second viewpoint image, where the location information represents a viewpoint location of the second viewpoint image; generating the preset quantity of to-be-processed virtual composite pixel matrices based on the adaptive convolution kernels and the pixels of the preset quantity of existing viewpoint images; and synthesizing the second viewpoint image by using the preset quantity of to-be-processed virtual composite pixel matrices. The method can improve efficiency and quality of synthesizing the second viewpoint image.

What is claimed is: 
     
         1 . A viewpoint image processing method comprising:
 obtaining at least two first viewpoint images, wherein the at least two first viewpoint images comprise images respectively captured at at least two viewpoint locations;   inputting the at least two first viewpoint images and location information of a to-be-synthesized second viewpoint image into a virtual viewpoint synthesis network, wherein the virtual viewpoint synthesis network is a network for synthesizing the second viewpoint image based on an adaptive convolution kernel corresponding to each pixel of the at least two first viewpoint images, the location information represents a viewpoint location of the second viewpoint image, the second viewpoint image is in a target area, and the target area comprises an area formed by the at least two first viewpoint images; and   obtaining the second viewpoint image through calculation by using the virtual viewpoint synthesis network.   
     
     
         2 . The method according to  claim 1 , wherein, the obtaining the second viewpoint image through calculation by using the virtual viewpoint synthesis network comprises:
 obtaining a geometric feature matrix between the at least two first viewpoint images, wherein the geometric feature matrix is a matrix used to represent information about a geometric location relationship between pixels of the at least two first viewpoint images;   generating the adaptive convolution kernel corresponding to each pixel of the at least two first viewpoint images based on the geometric feature matrix and the location information;   generating at least two to-be-processed virtual composite pixel matrices based on the adaptive convolution kernels and the pixels of the at least two first viewpoint images; and   synthesizing the second viewpoint image by using the at least two to-be-processed virtual composite pixel matrices.   
     
     
         3 . The method according to  claim 2 , wherein, the obtaining a geometric feature matrix between the at least two first viewpoint images comprises:
 extracting a feature from each of the at least two first viewpoint images to obtain at least two feature matrices;   performing a cross-correlation operation on every two of the at least two feature matrices to obtain one or more feature matrices after the operation; and   when one feature matrix after the operation is obtained, using the feature matrix after the operation as the geometric feature matrix, or, when a plurality of feature matrices after the operation are obtained, obtaining the geometric feature matrix through calculation based on the plurality of feature matrices after the operation.   
     
     
         4 . The method according to  claim 2 , wherein, the obtaining a geometric feature matrix between the at least two first viewpoint images comprises:
 extracting a pixel from each of the at least two first viewpoint images to obtain at least two pixel matrices;   composing the at least two pixel matrices into a hybrid pixel matrix; and   inputting the hybrid pixel matrix into a first preset convolutional neural network model to obtain the geometric feature matrix.   
     
     
         5 . The method according to  claim 2 , wherein, the location information of the to-be-synthesized second viewpoint image is coordinate values, and the generating the adaptive convolution kernel corresponding to each pixel of the at least two first viewpoint images based on the geometric feature matrix and the location information comprises:
 extending the coordinate values into a location matrix whose quantities of rows and columns are the same as those of the geometric feature matrix;   composing the location matrix and the geometric feature matrix into a hybrid information matrix;   inputting the hybrid information matrix into each of at least two second preset convolutional neural network models, wherein the at least two second preset convolutional neural network models have a same structure and different parameters; and   determining the adaptive convolution kernel corresponding to each pixel of the at least two first viewpoint images based on output results of the at least two second preset convolutional neural network models.   
     
     
         6 . The method according to  claim 2 , wherein, the generating at least two to-be-processed virtual composite pixel matrices based on the adaptive convolution kernels and the pixels of the at least two first viewpoint images comprises:
 performing convolution on the adaptive convolution kernel corresponding to each pixel of the at least two first viewpoint images and a pixel matrix with the pixel as a center in a one-to-one correspondence to obtain a virtual composite pixel corresponding to a pixel location of the pixel, wherein a quantity of rows of the pixel matrix is the same as that of the adaptive convolution kernel corresponding to the pixel, and a quantity of columns of the pixel matrix is the same as that of the adaptive convolution kernel corresponding to the pixel; and   composing the obtained virtual composite pixels into the at least two virtual composite pixel matrices.   
     
     
         7 . A viewpoint image processing device comprising:
 a processor;   a transceiver; and   a memory, wherein, the memory is configured to store a computer program and/or data, and the processor is configured to execute the computer program stored in the memory, such that the device performs the following operations:   obtaining at least two first viewpoint images, wherein the at least two first viewpoint images comprise images respectively captured at at least two viewpoint locations;   inputting the at least two first viewpoint images and location information of a to-be-synthesized second viewpoint image into a virtual viewpoint synthesis network, wherein the virtual viewpoint synthesis network is a network for synthesizing the second viewpoint image based on an adaptive convolution kernel corresponding to each pixel of the at least two first viewpoint images, the location information represents a viewpoint location of the second viewpoint image, the second viewpoint image is in a target area, and the target area comprises an area formed by the at least two first viewpoint images; and   obtaining the second viewpoint image through calculation by using the virtual viewpoint synthesis network.   
     
     
         8 . The device according to  claim 7 , wherein, the obtaining the second viewpoint image through calculation by using the virtual viewpoint synthesis network comprises:
 obtaining a geometric feature matrix between the at least two first viewpoint images, wherein the geometric feature matrix is a matrix used to represent information about a geometric location relationship between pixels of the at least two first viewpoint images;   generating the adaptive convolution kernel corresponding to each pixel of the at least two first viewpoint images based on the geometric feature matrix and the location information;   generating at least two to-be-processed virtual composite pixel matrices based on the adaptive convolution kernels and the pixels of the at least two first viewpoint images; and   synthesizing the second viewpoint image by using the at least two to-be-processed virtual composite pixel matrices.   
     
     
         9 . The device according to  claim 8 , wherein, the obtaining a geometric feature matrix between the at least two first viewpoint images comprises:
 extracting a feature from each of the at least two first viewpoint images to obtain at least two feature matrices;   performing a cross-correlation operation on every two of the at least two feature matrices to obtain one or more feature matrices after the operation; and   when one feature matrix after the operation is obtained, using the feature matrix after the operation as the geometric feature matrix, or, when a plurality of feature matrices after the operation are obtained, obtaining the geometric feature matrix through calculation based on the plurality of feature matrices after the operation.   
     
     
         10 . The device according to  claim 8 , wherein, the obtaining a geometric feature matrix between the at least two first viewpoint images comprises:
 extracting a pixel from each of the at least two first viewpoint images to obtain at least two pixel matrices;   composing the at least two pixel matrices into a hybrid pixel matrix; and   inputting the hybrid pixel matrix into a first preset convolutional neural network model to obtain the geometric feature matrix.   
     
     
         11 . The device according to  claim 8 , wherein, the location information of the to-be-synthesized second viewpoint image is coordinate values, and the generating the adaptive convolution kernel corresponding to each pixel of the at least two first viewpoint images based on the geometric feature matrix and the location information of the to-be-synthesized second viewpoint image comprises:
 extending the coordinate values into a location matrix whose quantities of rows and columns are the same as those of the geometric feature matrix;   composing the location matrix and the geometric feature matrix into a hybrid information matrix;   inputting the hybrid information matrix into each of at least two second preset convolutional neural network models, wherein the at least two second preset convolutional neural network models have a same structure and different parameters; and   determining the adaptive convolution kernel corresponding to each pixel of the at least two first viewpoint images based on output results of the at least two second preset convolutional neural network models.   
     
     
         12 . The device according to  claim 9 , wherein, the location information of the to-be-synthesized second viewpoint image is coordinate values, and the generating the adaptive convolution kernel corresponding to each pixel of the at least two first viewpoint images based on the geometric feature matrix and the location information of the to-be-synthesized second viewpoint image comprises:
 extending the coordinate values into a location matrix whose quantities of rows and columns are the same as those of the geometric feature matrix;   composing the location matrix and the geometric feature matrix into a hybrid information matrix;   inputting the hybrid information matrix into each of at least two second preset convolutional neural network models, wherein the at least two second preset convolutional neural network models have a same structure and different parameters; and   determining the adaptive convolution kernel corresponding to each pixel of the at least two first viewpoint images based on output results of the at least two second preset convolutional neural network models.   
     
     
         13 . The device according to  claim 10 , wherein, the location information of the to-be-synthesized second viewpoint image is coordinate values, and the generating the adaptive convolution kernel corresponding to each pixel of the at least two first viewpoint images based on the geometric feature matrix and the location information of the to-be-synthesized second viewpoint image comprises:
 extending the coordinate values into a location matrix whose quantities of rows and columns are the same as those of the geometric feature matrix;   composing the location matrix and the geometric feature matrix into a hybrid information matrix;   inputting the hybrid information matrix into each of at least two second preset convolutional neural network models, wherein the at least two second preset convolutional neural network models have a same structure and different parameters; and   determining the adaptive convolution kernel corresponding to each pixel of the at least two first viewpoint images based on output results of the at least two second preset convolutional neural network models.   
     
     
         14 . The device according to  claim 8 , wherein, the generating at least two to-be-processed virtual composite pixel matrices based on the adaptive convolution kernels and the pixels of the at least two first viewpoint images comprises:
 performing convolution on the adaptive convolution kernel corresponding to each pixel of the at least two first viewpoint images and a pixel matrix with the pixel as a center in a one-to-one correspondence to obtain a virtual composite pixel corresponding to a pixel location of the pixel, wherein a quantity of rows of the pixel matrix is the same as that of the adaptive convolution kernel corresponding to the pixel, and a quantity of columns of the pixel matrix is the same as that of the adaptive convolution kernel corresponding to the pixel; and   composing the obtained virtual composite pixels into the at least two virtual composite pixel matrices.   
     
     
         15 . The device according to  claim 9 , wherein, the generating at least two to-be-processed virtual composite pixel matrices based on the adaptive convolution kernels and the pixels of the at least two first viewpoint images comprises:
 performing convolution on the adaptive convolution kernel corresponding to each pixel of the at least two first viewpoint images and a pixel matrix with the pixel as a center in a one-to-one correspondence to obtain a virtual composite pixel corresponding to a pixel location of the pixel, wherein a quantity of rows of the pixel matrix is the same as that of the adaptive convolution kernel corresponding to the pixel, and a quantity of columns of the pixel matrix is the same as that of the adaptive convolution kernel corresponding to the pixel; and   composing the obtained virtual composite pixels into the at least two virtual composite pixel matrices.   
     
     
         16 . The device according to  claim 10 , wherein, the generating at least two to-be-processed virtual composite pixel matrices based on the adaptive convolution kernels and the pixels of the at least two first viewpoint images comprises:
 performing convolution on the adaptive convolution kernel corresponding to each pixel of the at least two first viewpoint images and a pixel matrix with the pixel as a center in a one-to-one correspondence to obtain a virtual composite pixel corresponding to a pixel location of the pixel, wherein a quantity of rows of the pixel matrix is the same as that of the adaptive convolution kernel corresponding to the pixel, and a quantity of columns of the pixel matrix is the same as that of the adaptive convolution kernel corresponding to the pixel; and   composing the obtained virtual composite pixels into the at least two virtual composite pixel matrices.   
     
     
         17 . The device according to  claim 11 , wherein, the generating at least two to-be-processed virtual composite pixel matrices based on the adaptive convolution kernels and the pixels of the at least two first viewpoint images comprises:
 performing convolution on the adaptive convolution kernel corresponding to each pixel of the at least two first viewpoint images and a pixel matrix with the pixel as a center in a one-to-one correspondence to obtain a virtual composite pixel corresponding to a pixel location of the pixel, wherein a quantity of rows of the pixel matrix is the same as that of the adaptive convolution kernel corresponding to the pixel, and a quantity of columns of the pixel matrix is the same as that of the adaptive convolution kernel corresponding to the pixel; and   composing the obtained virtual composite pixels into the at least two virtual composite pixel matrices.   
     
     
         18 . A non-transitory computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for image processing, and when the computer program is executed by one or more processors, the one or more processors performs operations comprising:
 obtaining at least two first viewpoint images, wherein the at least two first viewpoint images comprise images respectively captured at at least two viewpoint locations;   inputting the at least two first viewpoint images and location information of a to-be-synthesized second viewpoint image into a virtual viewpoint synthesis network, wherein the virtual viewpoint synthesis network is a network for synthesizing the second viewpoint image based on an adaptive convolution kernel corresponding to each pixel of the at least two first viewpoint images, the location information represents a viewpoint location of the second viewpoint image, the second viewpoint image is in a target area, and the target area comprises an area formed by the at least two first viewpoint images; and   obtaining the second viewpoint image through calculation by using the virtual viewpoint synthesis network.   
     
     
         19 . The non-transitory computer-readable storage medium according to  claim 18 , wherein, the obtaining the second viewpoint image through calculation by using the virtual viewpoint synthesis network comprises:
 obtaining a geometric feature matrix between the at least two first viewpoint images, wherein the geometric feature matrix is a matrix used to represent information about a geometric location relationship between pixels of the at least two first viewpoint images;   generating the adaptive convolution kernel corresponding to each pixel of the at least two first viewpoint images based on the geometric feature matrix and the location information;   generating at least two to-be-processed virtual composite pixel matrices based on the adaptive convolution kernels and the pixels of the at least two first viewpoint images; and   synthesizing the second viewpoint image by using the at least two to-be-processed virtual composite pixel matrices.   
     
     
         20 . The non-transitory computer-readable storage medium according to  claim 19 , wherein, the obtaining a geometric feature matrix between the at least two first viewpoint images comprises:
 extracting a feature from each of the at least two first viewpoint images to obtain at least two feature matrices;   performing a cross-correlation operation on every two of the at least two feature matrices to obtain one or more feature matrices after the operation; and   when one feature matrix after the operation is obtained, using the feature matrix after the operation as the geometric feature matrix, or, when a plurality of feature matrices after the operation are obtained, obtaining the geometric feature matrix through calculation based on the plurality of feature matrices after the operation.