US11417063B2ActiveUtilityA1

Determining a three-dimensional representation of a scene

58
Assignee: NVIDIA CORPPriority: Sep 1, 2020Filed: Feb 22, 2021Granted: Aug 16, 2022
Est. expirySep 1, 2040(~14.2 yrs left)· nominal 20-yr term from priority
Inventors:Yunzhi LinJonathan TremblayStephen Walter TyreeStanley Thomas Birchfield
G06T 17/10G06T 7/55G06T 15/10G06T 17/00G06T 7/75G06T 2207/20084G06T 2200/08G06T 19/003G06T 2207/10016
58
PatentIndex Score
0
Cited by
74
References
28
Claims

Abstract

One or more images (e.g., images taken from one or more cameras) may be received, where each of the one or more images may depict a two-dimensional (2D) view of a three-dimensional (3D) scene. Additionally, the one or more images may be utilized to determine a three-dimensional (3D) representation of a scene. This representation may help an entity navigate an environment represented by the 3D scene.

Claims

exact text as granted — not AI-modified
What is claimed is: 
     
       1. A method, comprising:
 receiving one or more images; and 
 processing the one or more images, using at least one neural network, to determine a three-dimensional (3D) representation of a scene by:
 processing, by the at least one neural network, camera location data in combination with the one or more images to provide a point cloud or point mesh, 
 processing, by the at least one neural network, the point cloud or point mesh in combination with the one or more images to determine primitives for objects within the scene, and 
 fitting 3D models to the objects within the scene, based on the primitives determined for the objects within the scene. 
 
 
     
     
       2. The method of  claim 1 , wherein the one or more images are captured utilizing one or more cameras mounted on or separately from an entity. 
     
     
       3. The method of  claim 1 , wherein each of the one or more images depicts a two-dimensional (2D) view of the scene. 
     
     
       4. The method of  claim 1 , wherein determining the 3D representation of the scene includes calculating a 3D reconstruction of the scene. 
     
     
       5. The method of  claim 4 , wherein the 3D reconstruction of the scene includes the point cloud or point mesh. 
     
     
       6. The method of  claim 1 , further comprising determining a camera pose for each of the one or more images. 
     
     
       7. The method of  claim 1 , wherein determining the 3D representation of the scene includes performing primitive shape segmentation within the scene which includes determining the primitives for the objects within the scene. 
     
     
       8. The method of  claim 1 , wherein for each of the one or more images, a 3D reconstruction of each scene for the image is used to render a virtual depth image for the scene. 
     
     
       9. The method of  claim 1 , wherein determining the 3D representation of the scene further includes performing object pose estimation for the objects within the scene. 
     
     
       10. The method of  claim 1 , further comprising utilizing the 3D representation of the scene to navigate an environment illustrated by the scene. 
     
     
       11. The method of  claim 1 , wherein processing the one or more images, using the at least one neural network, includes:
 processing, by a first neural network of the at least one neural network, the camera location data in combination with the one or more images to provide the point cloud or point mesh. 
 
     
     
       12. The method of  claim 11 , wherein the one or more images are processed using a plurality of neural networks, and wherein processing the one or more images, using the plurality of neural networks, includes:
 processing, by a second neural network of the plurality of neural networks, the point cloud or point mesh in combination with the one or more images to determine the primitives for the objects within the scene. 
 
     
     
       13. The method of  claim 12 , wherein fitting 3D models to the objects within the scene includes:
 for each of the primitives, comparing a plurality of predetermined parametric models to the primitive to identify a parametric model of the plurality of predetermined parametric models that matches the primitive. 
 
     
     
       14. The method of  claim 13 , wherein the parametric model that matches the primitive is adjusted to fit dimensions of the primitive. 
     
     
       15. The method of  claim 13 , wherein processing the one or more images, using the plurality of neural networks, includes:
 processing, by a third neural network of the plurality of neural networks, a set of known objects in combination with the one or more images to identify one or more instances of the known objects within the one or more images, and a pose of the one or more instances of the known objects identified within the one or more images. 
 
     
     
       16. The method of  claim 15 , wherein the one or more instances of the known objects identified within the one or more images and the pose of the one or more instances of the known objects identified within the one or more images are used in combination with the camera location data to determine a location of the one or more instances of the known objects within the scene. 
     
     
       17. The method of  claim 1 , wherein the 3D representation of the scene is utilized for robotic manipulation including:
 identification of rigid surfaces within the scene for obstacle avoidance. 
 
     
     
       18. The method of  claim 1 , wherein the 3D representation of the scene is utilized for robotic manipulation including:
 identification of instances of known objects within the scene for physical object grasping and manipulation. 
 
     
     
       19. A system comprising:
 a processor that is configured to: 
 receive one or more images; and 
 processing the one or more images, using at least one neural network, to determine a three-dimensional (3D) representation of a scene by:
 processing, by the at least one neural network, camera location data in combination with the one or more images to provide a point cloud or point mesh, 
 processing, by the at least one neural network, the point cloud or point mesh in combination with the one or more images to determine primitives for objects within the scene, and 
 fitting 3D models to the objects within the scene, based on the primitives determined for the objects within the scene. 
 
 
     
     
       20. The system of  claim 19 , wherein the one or more images are captured utilizing one or more cameras mounted on or separately from an entity. 
     
     
       21. The system of  claim 19 , wherein each of the one or more images depicts a two-dimensional (2D) view of the scene. 
     
     
       22. The system of  claim 19 , wherein determining the 3D representation of the scene includes calculating a 3D reconstruction of the scene. 
     
     
       23. The system of  claim 22 , wherein the 3D reconstruction of the scene includes the point cloud or point mesh. 
     
     
       24. The system of  claim 19 , further comprising determining a camera pose for each of the one or more images. 
     
     
       25. The system of  claim 19 , wherein determining the 3D representation of the scene includes performing primitive shape segmentation within the scene which includes determining the primitives for the objects within the scene. 
     
     
       26. A non-transitory computer-readable media storing computer instructions which when executed by one or more processors cause the one or more processors to perform a method comprising:
 receiving one or more images; and 
 processing the one or more images, using a neural network, to determine a three-dimensional (3D) representation of a scene by:
 processing, by the at least one neural network, camera location data in combination with the one or more images to provide a point cloud or point mesh, 
 processing, by the at least one neural network, the point cloud or point mesh in combination with the one or more images to determine primitives for objects within the scene, and 
 fitting 3D models to the objects within the scene, based on the primitives determined for the objects within the scene. 
 
 
     
     
       27. The non-transitory computer-readable media of  claim 26 , wherein the one or more images are received from one or more cameras, where the one or more cameras are mounted on or separately from an entity. 
     
     
       28. A method, comprising:
 receiving at an entity one or more images captured utilizing one or more cameras mounted on or separately from the entity; 
 processing the one or more images, using at least one neural network, to determine a three-dimensional (3D) representation of a scene, including:
 calculating, using the at least one neural network, a 3D reconstruction of the scene that includes a point cloud or point mesh, 
 performing primitive shape segmentation within the scene including processing, by the at least one neural network, the point cloud or point mesh in combination with the one or more images to determine primitives for objects within the scene, 
 fitting 3D models to the objects within the scene, based on the primitives determined for the objects within the scene, and 
 performing object pose estimation for one or more objects within the scene; and 
 
 utilizing the 3D representation of the scene by the entity to navigate an environment illustrated by the scene.

Cited by (0)

No later patents cite this yet.

References (0)

No backward citations on record.