System and method for reconstructing unoccupied 3d space

Abstract

The present teaching relates to method, system, medium, and implementations for understanding a three dimensional (3D) scene. Image data acquired by a camera at different time instances with respect to the 3D scene are received wherein the 3D scene includes a user or one or more objects. The face of the user is detected and tracked at different time instances. With respect to some of the time instances, a 2D user profile representing a region in the image data occupied by the user is generated based on a corresponding face detected and a corresponding 3D space in the 3D scene is estimated based on calibration parameters associated with the camera. Such estimated 3D space occupied by the user in the 3D scene is used to dynamically update a 3D space occupancy record of the 3D scene.

We claim: 
     
         1 . A method implemented on at least one machine including at least one processor, memory, and communication platform capable of connecting to a network for estimating occupancy of a three-dimensional (3D) scene, the method comprising:
 receiving continuous image data, acquired by at least one sensor from the 3D scene having at least one of a user and one or more objects therein, wherein the user is engaged in a human machine dialogue;   detecting, based on the continuous image data, the user and the one or more objects at different time instances and corresponding characteristics associated therewith;   estimating 3D occupancy dynamics of the 3D scene across the different time instances by, with respect to each of the different time instances:
 determining, a spatial relationship between the user and each of the one or more objects, a 3D volumetric space occupied by the user, and a 3D space occupancy of the 3D scene based on the 3D volumetric space and the spatial relationship with each of the one or more objects, and 
 constructing a 3D space occupancy record of the 3D scene associated with the time instance based on the 3D space occupancy, wherein 
   the 3D space occupancy records related to different time instances describe the 3D occupancy dynamics.   
     
     
         2 . The method of  claim 1 , wherein the step of detecting the user comprises:
 detecting, from the continuous image data, a face of the user;   tracking the face of the user appearing in different image frames of the continuous image data; and   generating, with respect to each of at least some of the different image frames at different time instances, a 2D profile of the user based on the face detected, wherein the 2D profile corresponds to a region in the image frame where the user is detected.   
     
     
         3 . The method of  claim 2 , wherein the step of determining the 3D volumetric space at the time instance comprises:
 determining a 3D position of the face of the user in the 3D scene corresponding to the region in the image frame;   obtaining, based on a face based human model, a 3D prism in the 3D scene corresponding to the 3D position; and   estimating the 3D volumetric space occupied by the user in the 3D scene based on the 3D prism.   
     
     
         4 . The method of  claim 1 , wherein the 3D occupancy records corresponding to the different time instances include information about a trajectory of the user in the 3D scene during the human machine dialogue. 
     
     
         5 . The method of  claim 1 , wherein the 3D occupancy dynamics are rendered on a device during the human machine dialogue. 
     
     
         6 . The method of  claim 4 , wherein the human machine dialogue is conducted based on the 3D occupancy dynamics. 
     
     
         7 . Machine readable and non-transitory medium having information recorded thereon for estimating occupancy of a three-dimensional (3D) scene, wherein the information, when read by the machine, causes the machine to perform the following:
 receiving continuous image data, acquired by at least one sensor from the 3D scene having at least one of a user and one or more objects therein, wherein the user is engaged in a human machine dialogue;   detecting, based on the continuous image data, the user and the one or more objects at different time instances and corresponding characteristics associated therewith;   estimating 3D occupancy dynamics of the 3D scene across the different time instances by, with respect to each of the different time instances:
 determining, a spatial relationship between the user and each of the one or more objects, a 3D volumetric space occupied by the user, and a 3D space occupancy of the 3D scene based on the 3D volumetric space and the spatial relationship with each of the one or more objects, and 
 constructing a 3D space occupancy record of the 3D scene associated with the time instance based on the 3D space occupancy, wherein 
   the 3D space occupancy records related to different time instances describe the 3D occupancy dynamics.   
     
     
         8 . The medium of  claim 7 , wherein the step of detecting the user comprises:
 detecting, from the continuous image data, a face of the user;   tracking the face of the user appearing in different image frames of the continuous image data; and   generating, with respect to each of at least some of the different image frames at different time instances, a 2D profile of the user based on the face detected, wherein the 2D profile corresponds to a region in the image frame where the user is detected.   
     
     
         9 . The medium of  claim 8 , wherein the step of determining the 3D volumetric space at the time instance comprises:
 determining a 3D position of the face of the user in the 3D scene corresponding to the region in the image frame;   obtaining, based on a face based human model, a 3D prism in the 3D scene corresponding to the 3D position; and   estimating the 3D volumetric space occupied by the user in the 3D scene based on the 3D prism.   
     
     
         10 . The medium of  claim 7 , wherein the 3D occupancy records corresponding to the different time instances include information about a trajectory of the user in the 3D scene during the human machine dialogue. 
     
     
         11 . The medium of  claim 7 , wherein the 3D occupancy dynamics are rendered on a device during the human machine dialogue. 
     
     
         12 . The medium of  claim 10 , wherein the human machine dialogue is conducted based on the 3D occupancy dynamics. 
     
     
         13 . A system for estimating occupancy of a three-dimensional (3D) scene, comprising:
 at least one sensor configured for acquiring continuous image data from the 3D scene having at least one of a user and one or more objects therein, wherein the user is engaged in a human machine dialogue;   an detection unit implemented using a processor and configured for detecting, based on the continuous image data, the user and the one or more objects at different time instances and corresponding characteristics associated therewith;   a 3D space occupancy estimator implemented using a processor and configured for estimating 3D occupancy dynamics of the 3D scene across the different time instances by, with respect to each of the different time instances:
 obtaining, a spatial relationship between the user and each of the one or more objects, a 3D volumetric space occupied by the user, and a 3D space occupancy of the 3D scene based on the 3D volumetric space and the spatial relationship with each of the one or more objects, and 
 constructing a 3D space occupancy record of the 3D scene associated with the time instance based on the 3D space occupancy, wherein 
   the 3D space occupancy records related to different time instances describe the 3D occupancy dynamics.   
     
     
         14 . The system of  claim 13 , wherein the 3D space occupancy estimator comprises:
 a face detection unit implemented using a processor and configured for detecting, from the continuous image data, a face of the user;   a face based human tracking unit implemented using a processor and configured for tracking the face of the user appearing in different image frames of the continuous image data; and   a human 3D occupancy estimator implemented using a processor and configured for generating, with respect to each of at least some of the different image frames at different time instances, a 2D profile of the user based on the face detected, wherein the 2D profile corresponds to a region in the image frame where the user is detected.   
     
     
         15 . The system of  claim 14 , wherein the human 3D occupancy estimator is further configured for obtaining the 3D volumetric space at the time instance by:
 determining a 3D position of the face of the user in the 3D scene corresponding to the region in the image frame;   obtaining, based on a face based human model, a 3D prism in the 3D scene corresponding to the 3D position; and   estimating the 3D volumetric space occupied by the user in the 3D scene based on the 3D prism.   
     
     
         16 . The system of  claim 13 , wherein the 3D occupancy records corresponding to the different time instances include information about a trajectory of the user in the 3D scene during the human machine dialogue. 
     
     
         17 . The system of  claim 13 , wherein the 3D occupancy dynamics are rendered on a device during the human machine dialogue. 
     
     
         18 . The system of  claim 16 , wherein the human machine dialogue is conducted based on the 3D occupancy dynamics.