System and method for reconstructing a 3D human body using compact kit of depth cameras

Abstract

The invention presents a system and a method for 3D human reconstruction using a compact kit of depth cameras. Instead of using complex and expensive devices as in traditional methods, the proposed system and method employs a simple, easy-to-install system to accurately collect the human body shape. The generated model is capable of moving thanks to a skeleton system simulating the human skeleton. The proposed system includes four blocks: Data Collection Block, Point Cloud Standardization Block, Human Digitization Block and Output Block. The proposed method includes five steps: Point Cloud Collecting, Point Cloud Filtering, Point Cloud Calibrating, Point Cloud Optimizing and 3D Human Model Generating.

The invention claimed is: 
     
       1. A system for 3D human reconstruction using a compact kit of depth cameras, including four blocks:
 a data collection block including a module of two depth cameras aims to capture 3D images of a digitized person; output of the data collection block is a raw point cloud data which is captured by the depth cameras, depth data obtained by the two cameras is passed to a deep learning network to extract positions of human joints in the depth data; then using this data to interpolate 3D positions of human joints in 3D space, a distance between cameras and the digitized person is set up carefully to ensure capturing full image of the person (from head to toe) (about ≥ 1.5 m), an angle between two cameras can be adjusted flexibly, after completing system installation, positions of two depth cameras are fixed for a 3D human reconstruction process; 
 a point cloud standardization block includes two modules: a point cloud filtering module and a point cloud calibrating module, in which:
 the point cloud filtering module is responsible for removing redundant point clouds in the 3D image (for example: point clouds of the ground, ceiling . . . ) and removing noise caused by camera devices; the point cloud filtering module comprises a pre-filter and a fine filter, In particular:
 the pre-filter is accountable for removing data of objects appearing around the digitized person and the ground; 
 the fine filter is accountable for removing points or clusters of points which stay close or stuck to human's point cloud, these points are caused by errors of point cloud receivers; 
 
 the point cloud calibrating module performs mathematical calculations of spatial transformation to move the obtained point cloud to a pre-defined coordinate system, being ready for a human digitization block, specifically, in the coordinate system, the origin is placed at the top of a head of the human point cloud; the y-axis is coincident with vertical direction of the digitized person, the x-axis is perpendicular to the y-axis and coincident with a standing direction of the digitized person, the z-axis is determined based on a Cartesian coordinate system rule; 
 
 the human digitization block includes two modules: a human parametric model module and a point cloud optimization module, in which:
 the human parametric model module is a model comprising shape control parameters such as height, weight, chest circumference, waist circumference parameters; the human parametric model module includes three main components: shape parameters; pose parameters; skeletal simulation, in particular:
 shape parameters are 10 parameters controlling and transforming the human body shape into different types (such as tall, short, fat, thin, . . . ); 
 pose parameters are joints parameters of a skeletal system which simulates the human body, the number of joints in the parametric model is 24 and the number of pose parameters is 72; 
 skeletal simulation is to simulate a bones system of the human body, including 24 joints (head, neck, right shoulder, left shoulder, . . . ), these joints are accountable for ensuring the movement of the human parametric model is similar to that of a real person; 
 
 the point cloud optimization module calculates shape control parameters obtained by the human parametric model module so that the shape generated after the optimization process approximates the shape of the point cloud, the point cloud optimization module includes two main components: pose optimization, shape optimization, firstly, length of bones and angle between joints of the human body are calculated by pose optimization, the main idea of this step is to minimize errors between length of bones, angle of joints in the parametric model and those in the calibrated point cloud, next, shape parameters of the parametric model are similarly calculated by shape optimization to find a set of parameters according to the calibrated point cloud, by this way, the optimization process is implemented by minimizing the objective function which represents errors between the parametric model and point cloud data, the output of this module is a set of model parameters of the human parametric model that approximates the real model shape, this set of parameters is passed to an output block to generate a movable 3D model representing the digitized person; 
 
 the output block: displaying the human body in 3D space with a pre-defined file format. 
 
     
     
       2. A method for 3D human reconstruction using a compact kit of depth cameras, including the following steps:
 step 1: collecting point cloud data, 
 in this step, point cloud data captured by the two depth cameras are collected and aggregated into one raw point cloud data, the point cloud aggregation is implemented by determining relative positions between cameras in the compact kit of depth cameras, which are calculated through an overlapping region of two cameras; 
 step 2: filtering point cloud data; 
 in this step, processing in turn pre-filter and fine-filter in a point cloud filtering module, in particular: 
 pre-filter: connecting positions of 3D joints to form a human skeleton having bones, with each bone in the skeleton, create a cylinder that has a same length as the bone and a defined radius (different bones of different body parts as arm, leg, . . . will have different radius), points standing outside the cylinder will be removed, thereby eliminating objects around the digitized person; 
 fine filter: using algorithms for statistical outlier removal, the point cloud is placed following a standard distribution, the expectation-maximization algorithm is then applied to estimate parameters of the statistical model, the final step is calculating a probability of elements belonging to the original data, elements with a low probability will be considered as outliners and removed; 
 output of this step is a point cloud that has been processed, containing only human data and removing irrelevant data already, this point cloud will be calibrated in the next step to suit the calibration step; 
 step 3: calibrating point cloud data; 
 point cloud calibration is a crucial step in preparing for the 3D human reconstruction process, the purpose here is to move the obtained point cloud to a standard coordinate system, specifically, spatial transformation algorithms are employed to move the origin to the top of the human head, the y-axis coincides with a vertical direction of human, the x-axis is perpendicular to the y-axis and coincides with a standing direction of human, the z-axis is determined based on a Cartesian coordinate system rule; 
 step 4: optimizing point cloud data; 
 3D joints data are used to determine relatively the length of bones and angles of joints in the real human body, from there, using interpolation to regress parameters of bones' length and parameters of pose that are relative to the real human body in the point cloud, pose parameters and shape parameters of the parametric model are initialized from the initial set of parameters and iteratively changed by the optimization algorithm to find a solution that minimizes below objective function: 
 
       
         
           
             
               L 
               = 
               
                 
                   1 
                   n 
                 
                 ⁢ 
                 
                   
                     ∑ 
                     i 
                   
                   
                     k 
                     ⁢ 
                     
                        
                       
                         
                           P 
                           SMPL 
                         
                         - 
                         
                           P 
                           NN 
                         
                       
                        
                     
                   
                 
               
             
           
         
         in which:
 P SMPL : a point staying on the parametric model; 
 P NN : a point on the point cloud closest to P SMPL ; 
 k: weight, k=1 if P SMPL  is inside the point cloud, k>1 if P SMPL , is outside the point cloud; 
 n: the number of sampling points on the parametric model; 
 
         to determine whether P SMPL  is inside or outside the point cloud data: 
         determine normal vector {right arrow over (n)} of each P SMPL  (the normal vector always points from inside model to outside), 
         determine direction vector {right arrow over (u)}=P SMPL −P NN , 
         if the angle between {right arrow over (u)} and {right arrow over (n)} is small than 180 degree, P SMPL  is inside the point cloud, otherwise P SMPL  is outside the point cloud; 
         step 5: generating 3D human model, 
         this step is executed in an output block, a human model is generated from results of digitizing process conforming to pre-defined rules, this model is fixed with the number of model vertex is 6890 and model polygon is 13444, besides, in order that the generated model is movable, a simulated skeletal system is generated appropriately for the generated model according to the rules of the human parametric model.