System and Method for Constructing a 3D Scene Model From an Image

Abstract

A system and method for constructing a 3D scene model comprising 3D objects and representing a scene, based upon a prior 3D scene model. The method comprises the steps of acquiring an image of the scene; initializing the computed 3D scene model to the prior 3D scene model; and modifying the computed 3D scene model to be consistent with the image. The step of modifying the computed 3D scene models consists of the sub-steps of comparing data of the image with objects of the 3D scene model, resulting in associated data and unassociated data; using the unassociated data to compute new objects that are not in the 3D scene model and adding the new objects to the 3D scene model; and using the associated data to detect objects in the prior 3D scene model that are absent and removing the absent objects from the 3D scene model. The present invention may also be used to construct multiple alternative 3D scene models.

What is claimed is: 
     
         1 . A method for moving a robotic manipulator in free space to avoid objects in a scene, based upon a prior 3D scene model corresponding to the scene in a prior state, the prior 3D scene model comprised of 3D object models corresponding to objects of the scene in the prior state, the method comprising the steps of:
 (a) receiving, by a processor, the current image of the scene;   (b) initializing, by the processor, the 3D scene model to the prior 3D scene model; and   (c) modifying the 3D scene model to be consistent with the image, by:
 (i) comparing, by the processor, data of the image with 3D object models of the 3D scene model, resulting in associated data and unassociated data; 
 (ii) using, by the processor, the unassociated data to compute new 3D object models and adding the new 3D object models to the 3D scene model; and 
 (iii) using, by the processor, the associated data to detect 3D object models in the prior 3D scene model that are absent and removing the absent 3D object models from the 3D scene model; 
   (d) computing, by the processor, a trajectory that avoids the 3D object models of the 3D scene model;   (e) directing, by the processor, the robotic manipulator to follow the computed trajectory in the scene; and   (f) following, by the robotic manipulator, the trajectory directed by the processor.   
     
     
         2 . The method of  claim 1 , wherein the step of computing new 3D object models further comprises requiring that a new 3D object model be in contact with one or more supporting 3D object models. 
     
     
         3 . The method of  claim 1 , wherein the step of computing new 3D object models further comprises requiring that a new 3D object model not be in collision with any other 3D object model. 
     
     
         4 . The method of  claim 1 , wherein the step of using the unassociated data to compute new 3D object models further comprises:
 (a) finding features in the unassociated data;   (b) matching the features in the unassociated data to features in a library of 3D object models;   (c) using the features in the library of 3D object models to identify 3D objects in the library of 3D object models.   
     
     
         5 . The method of  claim 1 , wherein the step of using the associated data to detect absent 3D object models further comprises:
 (a) computing synthetic features from the 3D scene model;   (b) comparing features of the associated data to the synthetic features to find 3D object models in the 3D scene model that do not appear in the image;   (c) subjecting the 3D object models in the 3D scene model that do not appear in the image to additional confirmation to determine whether the 3D object models are absent from the scene.   
     
     
         6 . A non-transitory computer readable storage medium having embodied thereon instructions for causing a computing device to execute a method for moving a robotic manipulator in free space to avoid objects in a scene, based upon a prior 3D scene model corresponding to the scene in a prior state, the prior 3D scene model comprised of 3D object models corresponding to objects of the scene in the prior state, the method comprising:
 (a) receiving the current image of the scene;   (b) initializing the 3D scene model to the prior 3D scene model; and   (c) modifying the 3D scene model to be consistent with the image, by:
 (i) comparing data of the image with 3D object models of the 3D scene model, resulting in associated data and unassociated data; 
 (ii) using the unassociated data to compute new 3D object models and adding the new 3D object models to the 3D scene model; and 
 (iii) using the associated data to detect 3D object models in the prior 3D scene model that are absent and removing the absent 3D object models from the 3D scene model; 
   (d) computing a trajectory that avoids the 3D object models of the 3D scene model; and   (e) directing the robotic manipulator to follow the computed trajectory in the scene.   
     
     
         7 . A method for grasping an object in a scene by a robotic manipulator, based upon and a prior 3D scene model corresponding to the scene in a prior state, the prior 3D scene model comprised of 3D object models corresponding to objects of the scene in the prior state, the method comprising the steps of:
 (a) receiving, by a processor, the current image of the scene;   (b) initializing, by the processor, the 3D scene model to the prior 3D scene model; and   (c) modifying the 3D scene model to be consistent with the image, by:
 (i) comparing, by the processor, data of the image with 3D object models of the 3D scene model, resulting in associated data and unassociated data; 
 (ii) using, by the processor, the unassociated data to compute new 3D object models and adding the new 3D object models to the 3D scene model; and 
 (iii) using, by the processor, the associated data to detect 3D object models in the prior 3D scene model that are absent and removing the absent 3D object models from the 3D scene model; 
   (d) computing, by the processor, a grasp of one of the 3D object models of the 3D scene model;   (e) directing, by the processor, the robotic manipulator to grasp the corresponding object in the scene using the computed grasp; and   (f) grasping, by the robotic manipulator, the object directed by the processor using the grasp directed by the processor.   
     
     
         8 . The method of  claim 7 , wherein the step of computing new 3D object models further comprises requiring that a new 3D object model be in contact with one or more supporting 3D object models. 
     
     
         9 . The method of  claim 7 , wherein the step of computing new 3D object models further comprises requiring that a new 3D object model not be in collision with any other 3D object model. 
     
     
         10 . The method of  claim 7 , wherein the step of using the unassociated data to compute new 3D object models further comprises:
 (a) finding features in the unassociated data;   (b) matching the features in the unassociated data to features in a library of 3D object models;   (c) using the features in the library of 3D object models to identify 3D objects in the library of 3D object models.   
     
     
         11 . The method of  claim 7 , wherein the step of using the associated data to detect absent 3D object models further comprises:
 (a) computing synthetic features from the 3D scene model;   (b) comparing features of the associated data to the synthetic features to find 3D object models in the 3D scene model that do not appear in the image;   (c) subjecting the 3D object models in the 3D scene model that do not appear in the image to additional confirmation to determine whether the 3D object models are absent from the scene.   
     
     
         12 . A non-transitory computer readable storage medium having embodied thereon instructions for causing a computing device to execute a method for grasping an object in a scene by a robotic manipulator, based upon a prior 3D scene model corresponding to the scene in a prior state, the prior 3D scene model comprised of 3D object models corresponding to objects of the scene in the prior state, the method comprising the steps of:
 (a) receiving, by a processor, the current image of the scene;   (b) initializing, by the processor, the 3D scene model to the prior 3D scene model; and   (c) modifying the 3D scene model to be consistent with the image, by:
 (i) comparing, by the processor, data of the image with 3D object models of the 3D scene model, resulting in associated data and unassociated data; 
 (ii) using, by the processor, the unassociated data to compute new 3D object models and adding the new 3D object models to the 3D scene model; and 
 (iii) using, by the processor, the associated data to detect 3D object models in the prior 3D scene model that are absent and removing the absent 3D object models from the 3D scene model; 
   (d) computing, by the processor, a grasp of one of the 3D object models of the 3D scene model; and   (e) directing, by the processor, the robotic manipulator to grasp the corresponding object in the scene using the computed grasp.   
     
     
         13 . A system for moving a robotic manipulator in free space to avoid objects in a scene, based upon a prior 3D scene model corresponding to the scene in a prior state, the prior 3D scene model comprised of 3D object models corresponding to objects of the scene in the prior state, the system comprising:
 (a) a processor configured to:
 (i) receive the current image of the scene; 
 (ii) initialize the 3D scene model to the prior 3D scene model; and 
 (iii) modify the 3D scene model to be consistent with the image, by:
 (A) comparing data of the image with the 3D object models of the 3D scene model, resulting in associated data and unassociated data; 
 (B) using the unassociated data to compute new 3D object models and adding the new 3D object models to the 3D scene model; 
 (C) using the associated data to detect 3D object models in the prior 3D scene model that are absent and removing the absent 3D object models from the 3D scene model; 
 
 (iv) compute a trajectory that avoids the 3D object models of the 3D scene model; and 
 (v) direct the robotic manipulator to follow the computed trajectory in the scene; and 
   (b) the robotic manipulator configured to follow a trajectory as directed by the processor.   
     
     
         14 . The system of  claim 13 , wherein the step of computing new 3D object models further comprises requiring that a new 3D object model be in contact with one or more supporting 3D object models. 
     
     
         15 . The system of  claim 13 , wherein the step of computing new 3D object models further comprises requiring that a new 3D object model not be in collision with any other 3D object model. 
     
     
         16 . The system of  claim 13 , wherein the step of using the unassociated data to compute new 3D object models further comprises:
 (a) finding features in the unassociated data;   (b) matching the features in the unassociated data to features in a library of 3D object models;   (c) using the features in the library of 3D object models to identify 3D objects in the library of 3D object models.   
     
     
         17 . The system of  claim 13 , wherein the step of using the associated data to detect absent 3D object models further comprises:
 (a) computing synthetic features from the 3D scene model;   (b) comparing features of the associated data to the synthetic features to find 3D object models in the 3D scene model that do not appear in the image;   (c) subjecting the 3D object models in the 3D scene model that do not appear in the image to additional confirmation to determine whether the 3D object models are absent from the scene.   
     
     
         18 . A system for grasping an object in a scene by a robotic manipulator, based upon a prior 3D scene model corresponding to the scene in a prior state, the prior 3D scene model comprised of 3D object models corresponding to objects of the scene in the prior state, the system comprising:
 (a) a processor configured to:
 (i) receive the current image of the scene; 
 (ii) initialize the 3D scene model to the prior 3D scene model; and 
 (iii) modify the 3D scene model to be consistent with the image, by:
 (A) comparing data of the image with the 3D object models of the 3D scene model, resulting in associated data and unassociated data; 
 (B) using the unassociated data to compute new 3D object models and adding the new 3D object models to the 3D scene model; 
 (C) using the associated data to detect 3D object models in the prior 3D scene model that are absent and removing the absent 3D object models from the 3D scene model; 
 
 (iv) compute a grasp of one of the 3D object models of the 3D scene model; and 
 (v) direct the robotic manipulator to grasp the corresponding object in the scene using the computed grasp; and 
   (b) the robotic manipulator configured to grasp an object as directed by the processor.   
     
     
         19 . The system of  claim 18 , wherein the step of computing new 3D object models further comprises requiring that a new 3D object model be in contact with one or more supporting 3D object models. 
     
     
         20 . The system of  claim 18 , wherein the step of computing new 3D object models further comprises requiring that a new 3D object model not be in collision with any other 3D object model. 
     
     
         21 . The system of  claim 18 , wherein the step of using the unassociated data to compute new 3D object models further comprises:
 (a) finding features in the unassociated data;   (b) matching the features in the unassociated data to features in a library of 3D object models;   (c) using the features in the library of 3D object models to identify 3D objects in the library of 3D object models.   
     
     
         22 . The system of  claim 18 , wherein the step of using the associated data to detect absent 3D object models further comprises:
 (a) computing synthetic features from the 3D scene model;   (b) comparing features of the associated data to the synthetic features to find 3D object models in the 3D scene model that do not appear in the image;   (c) subjecting the 3D object models in the 3D scene model that do not appear in the image to additional confirmation to determine whether the 3D object models are absent from the scene.