Evaluation of Three-Dimensional Scenes Using Two-Dimensional Representations

Abstract

A system adapted to implement a learning rule in a three-dimensional (3D) environment is described. The system includes: a renderer adapted to generate a two-dimensional (2D) image based at least partly on a 3D scene; a computational element adapted to generate a set of appearance features based at least partly on the 2D image; and an attribute classifier adapted to generate at least one set of learned features based at least partly on the set of appearance features and to generate a set of estimated scene features based at least partly on the set of learned features. A method labels each image from among the set of 2D images with scene information regarding the 3D scene; selects a set of learning modifiers based at least partly on the labeling of at least two images; and updates a set of weights based at least partly on the set of learning modifiers.

I claim: 
     
         1 . A robotic device that implements a learning rule in a three-dimensional (3D) environment, the robotic device comprising:
 a camera that captures and renders a plurality of two-dimensional (2D) images associated with a 3D environment;   a processor for executing a set of instructions; and   a non-transitory medium that stores the set of instructions, wherein the set of instructions comprises:
 generating a set of appearance features based at least partly on a 2D image from among the plurality of 2D images; 
 generating a set of learned features based at least partly on each set of appearance features; and 
 generating a set of estimated environment features based at least partly on the set of learned features. 
   
     
     
         2 . The robotic device of  claim 1 , the set of instructions further comprising:
 evaluating the set of estimated scene features and the learning rule; and   updating a first set of parameters used to generate the set of learned features.   
     
     
         3 . The robotic device of  claim 2 , the set of instructions further comprising:
 evaluating the set of estimated scene features and the learning rule; and   updating a second set of parameters used to generate the set of estimated scene features.   
     
     
         4 . The robotic device of  claim 3 , wherein:
 each learned feature in the set of learned features is calculated based at least partly on a non-linear function applied to a sum of cross products of a vector of appearance features and a 2D update matrix based at least partly on the first set of parameters; and   each estimated scene feature in the set of estimated scene features is calculated based at least partly on the non-linear function applied to a sum of cross products of a vector of learned features and a 2D update matrix based at least partly on the second set of parameters.   
     
     
         5 . The robotic device of  claim 3 , wherein the 3D environment is a virtual environment. 
     
     
         6 . The robotic device of  claim 5 , wherein the 3D environment comprises a set of true labels of scene features, and each of the first and second sets of parameters is based at least partly on the set of true labels. 
     
     
         7 . The robotic device of  claim 6 , wherein the set of true labels comprises a spatial map of Boolean values. 
     
     
         8 . An automated method that appends dense labels to two-dimensional (2D) images, the method comprising:
 recording a video comprising a sequence of 2D images related to a three-dimensional (3D) scene;   capturing a 2D image related to the 3D scene;   evaluating the video to identify information that is absent from the captured 2D image;   encoding the identified information into a file comprising the 2D image.   
     
     
         9 . The automated method of  claim 8  further comprising using the encoded information to predict a path of an object within the 3D scene. 
     
     
         10 . The automated method of  claim 8  further comprising using the encoded information to determine relative positions of a set of objects within the 3D scene. 
     
     
         11 . The automated method of  claim 8 , wherein the identified information is encoded using dense labels. 
     
     
         12 . The automated method of  claim 8 , wherein the 3D scene is associated with a virtual environment. 
     
     
         13 . The automated method of  claim 8 , wherein the 3D scene is associated with a physical environment and the recording, capturing, evaluating, and encoding are performed by a robotic device having at least one 2D camera and at least one processor. 
     
     
         14 . An automated method that predicts image information for at least one image in a sequence of images, the method comprising:
 determining a feature response for a first image in the sequence of images;   determining a feature response for a second image in the sequence of images;   identifying a transform based on the feature responses; and   applying the transform to the second image to at least partly predict a third image in the sequence of images.   
     
     
         15 . The automated method of  claim 14  further comprising:
 identifying mid-level features invariant to the transform; and 
 determining a space of encoded activity based on the transform. 
 
     
     
         16 . The automated method of  claim 15  further comprising representing a joint probability of the transform and the space. 
     
     
         17 . The automated method of  claim 14  further comprising:
 aligning the first image to the second image; 
 calculating a difference between the first image and the second image; and 
 predicting properties of the third image based at least partly on the calculated difference. 
 
     
     
         18 . The automated method of  claim 17  further comprising:
 calculating edge discontinuities in a motion field associated with the aligned images; and 
 predicting properties of the third image based at least partly on the calculated edge discontinuities. 
 
     
     
         19 . The automated method of  claim 14 , wherein the sequence of images is associated with a three-dimensional scene. 
     
     
         20 . The automated method of  claim 19 , wherein the sequences of images comprises a set of two-dimensional images captured over time.