Method and apparatus for matching user motion tracked by heterogeneous sensors

Abstract

A method and an apparatus for registering motion of a user tracked by heterogeneous sensors are disclosed. The method includes initializing the first sensor so that an origin of a first sensor coordinate system is adjacent to an origin of a device coordinate system, wherein the device coordinate system is a three-dimensional coordinate system with a current position of the device as the origin; and determining a calibration parameter for transforming coordinates of the plurality of feature points according to the first sensor coordinate system to coordinates according to a second sensor coordinate system, based on the position and orientation of the device in the second sensor coordinate system.

What is claimed is: 
     
         1 . A method for registering motion of a user tracked by heterogeneous sensors, the heterogeneous sensors comprising a first sensor for detecting a plurality of feature points representing joints of the user and a second sensor for detecting a position and an orientation of a device carried by the user, the method comprising:
 initializing the first sensor so that an origin of a first sensor coordinate system is adjacent to an origin of a device coordinate system, wherein the device coordinate system is a three-dimensional coordinate system with a current position of the device as the origin; and   determining a calibration parameter for transforming coordinates of the plurality of feature points according to the first sensor coordinate system to coordinates according to a second sensor coordinate system, based on the position and orientation of the device in the second sensor coordinate system.   
     
     
         2 . The method of  claim 1 , further comprising:
 registering a user motion acquired from the first sensor with a user motion acquired from the second sensor using the calibration parameter.   
     
     
         3 . The method of  claim 2 , wherein the device comprises an HMD headset,
 wherein the registering the user motion comprises:   receiving data for tracking a head motion of the user from the first sensor by detecting a feature point corresponding to a head joint of the user from the plurality of feature points;   receiving data for tracking the head motion of the user from the second sensor by detecting a position and an orientation of the HMD headset; and   registering the head motion using the calibration parameter.   
     
     
         4 . The method of  claim 3 , wherein the device further comprises an XR controller,
 wherein the registering the user motion comprises:   receiving data for tracking a hand motion of the user from the first sensor by detecting a feature point corresponding to a hand joint of the user from the plurality of feature points;   receiving data for tracking the hand motion of the user from the second sensor by detecting a position and an orientation of the XR controller; and   registering the hand motion using the calibration parameter.   
     
     
         5 . The method of  claim 4 , further comprising:
 displaying a three-dimensional avatar for interacting with metaverse content to the user based on the registered user motion.   
     
     
         6 . The method of  claim 2 , further comprising:
 displaying a three-dimensional avatar for interacting with metaverse content to the user based on the registered user motion.   
     
     
         7 . The method of  claim 3 , further comprising:
 displaying a three-dimensional avatar for interacting with metaverse content to the user based on the registered user motion.   
     
     
         8 . The method of  claim 1 , wherein the first sensor is an RGB-D camera, and the second sensor is an outside-in HID tracker comprising at least one base station. 
     
     
         9 . The method of  claim 1 , further comprising updating the calibration parameter,
 wherein the updating of the calibration parameter comprises:   acquiring first data representing the position and the orientation of the device by using the first sensor based on the calibration parameter;   acquiring second data representing the position and the orientation of the device by using the second sensor; and   adjusting the calibration parameter based on a difference between the first data and the second data.   
     
     
         10 . The method of  claim 9 , wherein the calibration parameter is updated for each frame. 
     
     
         11 . The method of  claim 9 , wherein the calibration parameter is updated for each preset period or preset time points. 
     
     
         12 . The method of  claim 1 , wherein the calibration parameter comprises a translation matrix and a rotation matrix for transforming coordinates according to the first sensor coordinate system to coordinates according to the second sensor coordinate system,
 wherein the determining the calibration parameter comprises:   determining the translation matrix using the position of the device in the second sensor coordinate system; and   determining the rotation matrix using the orientation of the device in the second sensor coordinate system.   
     
     
         13 . The method of  claim 12 , further comprising updating the translation matrix,
 wherein the updating of the translation matrix comprises:   acquiring data representing a position of a feature point corresponding to the device from the plurality of feature points in the first sensor coordinate system, by using the first sensor;   acquiring first data representing the position of the feature point corresponding to the device in the second sensor coordinate system, by using the calibration parameter;   acquiring second data representing the position of the device in the second sensor coordinate system, by using the second sensor; and   adjusting the translation matrix based on a difference between the first data and the second data.   
     
     
         14 . The method of  claim 12 , further comprising updating the rotation matrix,
 wherein the updating of the rotation matrix comprises:   acquiring first data representing a direction of a feature point corresponding to the device from the plurality of feature points, by using the first sensor;   acquiring second data representing the orientation of the device, by using the second sensor; and   adjusting the rotation matrix based on a difference between the first data and the second data.   
     
     
         15 . An apparatus for registering motion of a user tracked by heterogeneous sensors, the heterogeneous sensors comprising a first sensor for detecting a plurality of feature points representing joints of the user and a second sensor for detecting a position and an orientation of a device carried by the user, the apparatus comprising:
 at least one memory configured to store instructions; and   at least one processor,   wherein the at least one processor executes the instructions to:   initialize the first sensor so that an origin of a first sensor coordinate system is adjacent to an origin of a device coordinate system, wherein the device coordinate system is a three-dimensional coordinate system with a current position of the device as the origin; and   determine a calibration parameter for transforming coordinates of the plurality of feature points according to the first sensor coordinate system to coordinates according to a second sensor coordinate system, based on the position and orientation of the device in the second sensor coordinate system.