Motion tracking with multiple 3d cameras

Abstract

A system comprising at least two three-dimensional (3D) cameras that are each configured to produce a digital image with a depth value for each pixel of the digital image; and a processor configured to: perform inter-camera calibration by: (i) estimating a pose of a subject, based, at least in part, on a skeleton representation of a subject captured each of by said at least two 3D cameras, wherein said skeleton representation identifies a plurality of skeletal joints of said subject, and (ii) enhancing the estimated pose based, at least in part, on a 3D point cloud of a scene containing the subject, as captured by each of said at least two 3D cameras, and perform data merging of digital images captured by said at least two 3D cameras, wherein the data merging is per each of said identifications.

What is claimed is: 
     
         1 . A system comprising:
 at least two three-dimensional (3D) cameras that are each configured to produce a digital image with a depth value for each pixel of the digital image;   at least one hardware processor; and   a non-transitory computer-readable storage medium having stored thereon program instructions, the program instructions executable by the at least one hardware processor to:
 perform inter-camera calibration by: 
 (i) estimating a pose of a subject, based, at least in part, on a skeleton representation of the subject captured by each of said at least two 3D cameras, wherein said skeleton representation identifies a plurality of skeletal joints of said subject, and wherein each of said identifications has a confidence score and 
 (ii) enhancing the estimated pose based, at least in part, on a 3D point cloud of a scene containing the subject, as captured by each of said at least two 3D cameras, wherein said skeleton representation identifies a plurality of skeletal joints of said subject, and 
 perform data merging of digital images captured by said at least two 3D cameras, wherein the data merging is per each of said identification. 
   
     
     
         2 . The system of  claim 1 , wherein said data merging is a weighted data merging based, at least in part, on weights assigned to each of said joints, and wherein said weights are assigned based on at least some of: said confidence score, a frontality measure, and a distance measure. 
     
     
         3 . The system of  claim 1 , wherein said estimating comprises calculating an optimal 3D rigid transformation between each of said skeleton representations, and wherein said transformation is further optimized using random sample consensus (RANSAC). 
     
     
         4 . The system of  claim 1 , wherein said enhancing comprises aligning said 3D point cloud using an iterative closest point (ICP) algorithm. 
     
     
         5 . The system of  claim 1 , wherein each of said 3D point clouds further comprises 3D points representing a ground plane near the legs of said subject. 
     
     
         6 . The system of  claim 1 , comprising at least 3 said 3D cameras, wherein the program instructions are further executable to perform said inter-camera calibration with respect to each pair of neighboring said 3D cameras. 
     
     
         7 . A method comprising:
 performing inter-camera calibration with respect to at least two three-dimensional (3D) cameras that are each configured to produce a digital image with a depth value for each pixel of the digital image, by:
 (i) estimating a pose of a subject, based, at least in part, on a skeleton representation of the subject captured by each of said at least two 3D cameras, wherein said skeleton representation identifies a plurality of skeletal joints of said subject, and 
 (ii) enhancing the estimated pose based, at least in part, on a 3D point cloud of a scene containing the subject, as captured by each of said at least two 3D cameras, and 
   performing data merging of digital images captured by each of said at least two 3D cameras, wherein the data merging is per each of said identifications.   
     
     
         8 . The method of  claim 7 , wherein said data merging is a weighted data merging based, at least in part, on weights assigned to each of said joints, and wherein said weights are assigned based on at least some of: said confidence score, a frontality measure, and a distance measure. 
     
     
         9 . The method of  claim 7 , wherein said estimating comprises calculating an optimal 3D rigid transformation between each of said skeleton representations, and wherein said transformation is further optimized using random sample consensus (RANSAC). 
     
     
         10 . The method of  claim 7 , wherein said enhancing comprises aligning said 3D point cloud using an iterative closest point (ICP) algorithm. 
     
     
         11 . The method of  claim 7 , wherein each of said 3D point clouds further comprises 3D points representing a ground plane near the legs of said subject. 
     
     
         12 . A computer program product comprising a non-transitory computer-readable storage medium having program instructions embodied therewith, the program instructions executable by at least one hardware processor to:
 perform inter-camera calibration with respect to at least two three-dimensional (3D) cameras that are each configured to produce a digital image with a depth value for each pixel of the digital image, by:
 (iii) estimating a pose of a subject, based, at least in part, on a skeleton representation of the subject captured by each of said at least two 3D cameras, wherein said skeleton representation identifies a plurality of skeletal joints of said subject, and 
 (iv) enhancing the estimated pose based, at least in part, on a 3D point cloud of a scene containing the subject, as captured by each of said at least two 3D cameras, and 
   perform data merging of digital images captured by said at least two 3D cameras, wherein the data merging is per each of said identifications.   
     
     
         13 . The computer program product of  claim 12 , wherein said data merging is a weighted data merging based, at least in part, on weights assigned to each of said joints, and wherein said weights are assigned based on at least some of: said confidence score, a frontality measure, and a distance measure. 
     
     
         14 . The computer program product of  claim 12 , wherein said estimating comprises calculating an optimal 3D rigid transformation between each of said skeleton representations, and wherein said transformation is further optimized using random sample consensus (RANSAC). 
     
     
         15 . The computer program product of  claim 12 , wherein said enhancing comprises aligning said 3D point cloud using an iterative closest point (ICP) algorithm. 
     
     
         16 . The computer program product of  claim 12 , wherein each of said 3D point clouds further comprises 3D points representing a ground plane near the legs of said subject. 
     
     
         17 . The computer program product of  claim 12 , comprising at least 3 said 3D cameras, wherein the program instructions are further executable to perform said inter-camera calibration with respect to each pair of neighboring said 3D cameras.