Neural network-based picture filtering, coding, and decoding methods and apparatuses, device, and storage medium

Abstract

This application provides neural network-based picture filtering method performed by a computer device. The method includes the following operations: determining input information of a neural network loop filter (NNLF) for a target picture; and filtering the input information through the NNLF to obtain a filtered picture of the target picture, the NNLF including a plurality of convolution layers configured to extract picture feature information, and the plurality of convolution layers being obtained by processing a standard convolution layer in at least two of a multi-level receptive field (MLRF) mode, a tensor decomposition mode, and a group convolution (GC) mode. Thus, the filtering operation complexity of the NNLF is reduced without reducing the filtering performance of the NNLF, thereby improving the picture filtering effect.

What is claimed is: 
     
         1 . A neural network-based picture filtering method performed by an electronic device, the method comprising:
 determining input information of a neural network loop filter (NNLF) for a target picture; and   filtering the input information through the NNLF to obtain a filtered picture of the target picture, wherein the NNLF comprises a plurality of convolution layers configured to extract picture feature information; and the plurality of convolution layers are obtained by processing a standard convolution layer in at least two of a multi-level receptive field (MLRF) mode, a tensor decomposition mode, and a group convolution (GC) mode.   
     
     
         2 . The method according to  claim 1 , wherein the NNLF comprises a residual unit, the residual unit comprises a plurality of residual blocks (RBs) sequentially connected, an i th  RB in the plurality of RBs comprises all or part of the plurality of convolution layers, and i is a positive integer. 
     
     
         3 . The method according to  claim 2 , wherein the i th  RB comprises a first convolution unit, a second convolution unit, and a third convolution unit, and the first convolution unit and the second convolution unit are connected in parallel and then connected in series to the third convolution unit; and
 the first convolution unit comprises a first convolution layer, the second convolution unit comprises a second convolution layer, and the third convolution unit comprises a third convolution layer; convolution kernel sizes of the first convolution layer and the second convolution layer are different, and at least one of the first convolution layer, the second convolution layer, and the third convolution layer comprises a GC layer; the GC layer is configured to divide, in the GC mode, input feature information of the GC layer into a plurality of groups based on the number of channels, perform a convolution operation on each group of feature information, and then merge the number of channels as output information of the GC layer.   
     
     
         4 . The method according to  claim 3 , wherein for a j th  convolution layer in the first convolution layer, the second convolution layer, and the third convolution layer, the j th  convolution layer comprises: a first group sub-convolution layer and a second group sub-convolution layer obtained by performing, in the tensor decomposition mode and the GC mode, decomposition and channel grouping on a convolution layer having a convolution kernel size of m×n, m and n being both positive integers greater than 2, and j being a positive integer. 
     
     
         5 . The method according to  claim 4 , wherein a convolution kernel size of the first group sub-convolution layer is 1×n, and a convolution kernel size of the second group sub-convolution layer is m×1. 
     
     
         6 . The method according to  claim 3 , wherein the third convolution unit further comprises a fourth convolution layer, and the fourth convolution layer is connected in series to the third convolution layer. 
     
     
         7 . The method according to  claim 6 , wherein the fourth convolution layer comprises the GC layer; or
 the fourth convolution layer comprises: a third sub-convolution layer and a fourth sub-convolution layer obtained by decomposing, in the tensor decomposition mode, a convolution layer having a convolution kernel size of p×q, p and q being both positive integers greater than 2; or   the fourth convolution layer comprises: a third group sub-convolution layer and a fourth group sub-convolution layer obtained by performing, in the tensor decomposition mode and the GC mode, decomposition and channel grouping on the convolution layer having the convolution kernel size of p×q.   
     
     
         8 . The method according to  claim 2 , wherein the i th  RB comprises N convolution layers connected in series, and a kth convolution layer in the N convolution layers comprises: a fifth group sub-convolution layer and a sixth group sub-convolution layer obtained by performing, in the tensor decomposition mode and the GC mode, decomposition and channel grouping on a convolution layer having a convolution kernel size of r×s, r and s being both positive integers greater than 2, N being a positive integer, and k being a positive integer less than or equal to N. 
     
     
         9 . The method according to  claim 2 , wherein the NNLF further comprises: a shallow feature extraction unit, wherein the shallow feature extraction unit comprises at least one convolution layer; and
 the shallow feature extraction unit is configured to extract shallow feature information of the input information and input the shallow feature information into the residual unit.   
     
     
         10 . The method according to  claim 2 , wherein the NNLF further comprises: a feature mapping unit, wherein the feature mapping unit comprises at least one convolution layer; and
 the feature mapping unit is configured to map the picture feature information outputted by the residual unit to obtain the filtered picture.   
     
     
         11 . The method according to  claim 2 , wherein the residual unit comprises a luminance component residual unit and a chrominance component residual unit;
 the luminance component residual unit is configured to perform feature extraction on a luminance component of the target picture to obtain picture feature information of the luminance component of the target picture; and   the chrominance component residual unit is configured to perform feature extraction on a chrominance component of the target picture to obtain picture feature information of the chrominance component of the target picture.   
     
     
         12 . The method according to  claim 11 , wherein the NNLF further comprises a luminance component feature mapping unit connected to the luminance component residual unit, and a chrominance component feature mapping unit connected to the chrominance component residual unit;
 the luminance component feature mapping unit is configured to map picture luminance feature information outputted by the luminance component residual unit; and   the chrominance component feature mapping unit is configured to map picture chrominance feature information outputted by the chrominance component residual unit.   
     
     
         13 . The method according to  claim 1 , wherein the target picture is a reconstructed picture, and the method further comprises:
 inputting the filtered picture of the reconstructed picture into a display device for display, and saving the filtered picture of the reconstructed picture into a decoded picture buffer as a reference picture of another picture; or inputting the filtered picture of the reconstructed picture into the display device for presentation, and not using the filtered picture as the reference picture.   
     
     
         14 . An electronic device, comprising a processor and a memory,
 the memory being configured to store a computer program; and   the processor being configured to execute the computer program to implement a neural network-based picture filtering method including:   determining input information of a neural network loop filter (NNLF) for a target picture; and   filtering the input information through the NNLF to obtain a filtered picture of the target picture, wherein the NNLF comprises a plurality of convolution layers configured to extract picture feature information; and the plurality of convolution layers are obtained by processing a standard convolution layer in at least two of a multi-level receptive field (MLRF) mode, a tensor decomposition mode, and a group convolution (GC) mode.   
     
     
         15 . The electronic device according to  claim 14 , wherein the NNLF comprises a residual unit, the residual unit comprises a plurality of residual blocks (RBs) sequentially connected, an i th  RB in the plurality of RBs comprises all or part of the plurality of convolution layers, and i is a positive integer. 
     
     
         16 . The electronic device according to  claim 15 , wherein the NNLF further comprises: a shallow feature extraction unit, wherein the shallow feature extraction unit comprises at least one convolution layer; and
 the shallow feature extraction unit is configured to extract shallow feature information of the input information and input the shallow feature information into the residual unit.   
     
     
         17 . The electronic device according to  claim 15 , wherein the NNLF further comprises: a feature mapping unit, wherein the feature mapping unit comprises at least one convolution layer; and
 the feature mapping unit is configured to map the picture feature information outputted by the residual unit to obtain the filtered picture.   
     
     
         18 . The electronic device according to  claim 15 , wherein the residual unit comprises a luminance component residual unit and a chrominance component residual unit;
 the luminance component residual unit is configured to perform feature extraction on a luminance component of the target picture to obtain picture feature information of the luminance component of the target picture; and   the chrominance component residual unit is configured to perform feature extraction on a chrominance component of the target picture to obtain picture feature information of the chrominance component of the target picture.   
     
     
         19 . The electronic device according to  claim 14 , wherein the target picture is a reconstructed picture, and the method further comprises:
 inputting the filtered picture of the reconstructed picture into a display device for display, and saving the filtered picture of the reconstructed picture into a decoded picture buffer as a reference picture of another picture; or inputting the filtered picture of the reconstructed picture into the display device for presentation, and not using the filtered picture as the reference picture.   
     
     
         20 . A non-transitory computer-readable storage medium storing a computer program therein,
 the computer program, when executed by a processor of an electronic device, causing the electronic device to perform a neural network-based picture filtering method including:   determining input information of a neural network loop filter (NNLF) for a target picture; and   filtering the input information through the NNLF to obtain a filtered picture of the target picture, wherein the NNLF comprises a plurality of convolution layers configured to extract picture feature information; and the plurality of convolution layers are obtained by processing a standard convolution layer in at least two of a multi-level receptive field (MLRF) mode, a tensor decomposition mode, and a group convolution (GC) mode.