Rendering two-dimensional image of a dynamic three-dimensional scene

Abstract

Embodiments for rendering a 2D image of a dynamic 3D scene from different view angles using a neural radiance field (NeRF) are provided. In this regard, an AI image processing system is configured to process coordinates of a point in a dynamic 3D scene with an RNN over a number of time steps indicated by a time instance of interest to produce motion information of the point at the time instance of interest, process the motion information with a fully connected neural network to produce a displacement of the point from the coordinates in the dynamic 3D scene, and process a displaced point from a view angle of interest with the NeRF to render the point on the 2D image of the dynamic 3D scene at the time instance of interest. The displaced point is generated based on an estimate of the displacement of the point by leveraging motion cues.

What is claimed is: 
     
         1 . An artificial intelligence (AI) image processing system employing a neural radiance field (NeRF) to render a two-dimensional (2D) image of a dynamic three-dimensional (3D) scene from different view angles and different instances of time based on an implicit representation of the 3D scene, the AI image processing system comprising: at least one processor and a memory having instructions stored thereon that cause the at least one processor of the AI image processing system to:
 process coordinates of a point in a dynamic 3D scene with a recurrent neural network over a number of time steps indicated by a time instance of interest to produce motion information of the point at the time instance of interest;   process the motion information with a fully connected neural network to produce a displacement of the point from the coordinates in the dynamic 3D scene; and   process a displaced point from a view angle of interest with the NeRF trained for a static 3D scene to render the point on the 2D image of the dynamic 3D scene of the time instance of interest, wherein the displaced point is generated based on the displacement of the point.   
     
     
         2 . The AI image processing system of  claim 1 , wherein the recurrent neural network and the fully connected neural network are executed recursively for different instances of time until the time instance of interest is reached, such that a displacement of the point determined during a current iteration is an input to a subsequent iteration. 
     
     
         3 . The AI image processing system of  claim 1 , wherein the recurrent neural network includes gated recurrent units (GRUs), and wherein execution of each of the GRUs corresponds to a motion of the point at a time step. 
     
     
         4 . The AI image processing system of  claim 3 , wherein
 the recurrent neural network includes a fixed number of iterations of the gated recurrent units (GRUs) defining a maximum length of a prediction horizon, and   the recurrent neural network and the fully connected neural network are executed recursively for different instances of time until the time instance of interest is reached, such that a displacement of the point determined during a current iteration is an input to a subsequent iteration.   
     
     
         5 . The AI image processing system of  claim 4 , wherein each recursion operates on a time horizon with a length not greater than the maximum length of the prediction horizon. 
     
     
         6 . The AI image processing system of  claim 4 , wherein the displacement of the point determined during the current iteration is input to the subsequent iteration and is combined with a hidden state at the subsequent iteration using a non-linear mapping. 
     
     
         7 . The AI image processing system of  claim 4 , wherein an output at the current iteration is a function of a current input and a hidden state of the GRUs. 
     
     
         8 . The AI image processing system of  claim 4 , wherein the output of the GRUs at the current iteration is a non-linear function of a weighted sum of a current input and a hidden state at the current iteration. 
     
     
         9 . The AI image processing system of  claim 1 , wherein, the NeRF is trained to:
 implicitly represent 3D geometry and appearance of the 3D scene; and   learn a continuous 3D function with learnable parameters based on the view angle of interest and coordinates of the displaced point.   
     
     
         10 . The AI image processing system of  claim 1 , wherein the NeRF is trained using backpropagation through time (BPTT). 
     
     
         11 . The AI image processing system of  claim 1 , wherein the recurrent neural network (RNN) is a bi-directional RNN. 
     
     
         12 . The AI image processing system of  claim 1 , wherein rendering the point on the 2D image of the dynamic 3D scene produces a new view of the 3D scene. 
     
     
         13 . The AI image processing system of  claim 1 , wherein the point represents a location of an object in a 3D scene. 
     
     
         14 . A method for artificial intelligence (AI)-based image processing, the method being implemented by an AI image processing system employing a neural radiance field (NeRF) to render a two-dimensional (2D) image of a dynamic three-dimensional (3D) scene from different view angles and different instances of time based on an implicit representation of the 3D scene, the method comprising:
 processing coordinates of a point in a dynamic 3D scene with a recurrent neural network over a number of time steps indicated by a time instance of interest to produce motion information of the point at the time instance of interest;   processing the motion information with a fully connected neural network to produce a displacement of the point from the coordinates in the dynamic 3D scene; and   processing a displaced point from a view angle of interest with the NeRF trained for a static 3D scene to render the point on the 2D image of the dynamic 3D scene of the time instance of interest, wherein the displaced point is generated based on the displacement of the point.   
     
     
         15 . The method of  claim 14 , wherein the recurrent neural network and the fully connected neural network are executed recursively for different instances of time until the time instance of interest is reached, such that a displacement of the point determined during a current iteration is an input to a subsequent iteration. 
     
     
         16 . The method of  claim 14 , wherein the recurrent neural network includes gated recurrent units (GRUs), and wherein execution of each of the GRUs corresponds to a motion of the point at a time step. 
     
     
         17 . The method of  claim 16 , wherein
 the recurrent neural network includes a fixed number of iterations of the gated recurrent units (GRUs) defining a maximum length of a prediction horizon, and   the recurrent neural network and the fully connected neural network are executed recursively for different instances of time until the time instance of interest is reached, such that a displacement of the point determined during a current iteration is an input to a subsequent iteration.   
     
     
         18 . The method of  claim 17 , wherein each recursion operates on a time horizon with a length not greater than the maximum length of the prediction horizon. 
     
     
         19 . The method of  claim 14 , wherein rendering the point on the 2D image of the dynamic 3D scene produces a new view of the 3D scene. 
     
     
         20 . A non-transitory computer readable storage medium embodied thereon a program executable by a processor for performing a method, the method comprises:
 processing coordinates of a point in a dynamic 3D scene with a recurrent neural network over a number of time steps indicated by a time instance of interest to produce motion information of the point at the time instance of interest, wherein a neural radiance field (NeRF) is employed to render a two-dimensional (2D) image of a dynamic three-dimensional (3D) scene from different view angles and different instances of time based on an implicit representation of the 3D scene;   processing the motion information with a fully connected neural network to produce a displacement of the point from the coordinates in the dynamic 3D scene; and   processing a displaced point from a view angle of interest with the NeRF trained for a static 3D scene to render the point on the 2D image of the dynamic 3D scene of the time instance of interest, wherein the displaced point is generated based on the displacement of the point.