6DoF INSIDE-OUT TRACKING GAME CONTROLLER INITIAL REGISTRATION

Abstract

Methods and apparatus are provided for 6DoF inside-out tracking game control. In one novel aspect, a multi-processor architecture is used for VI-SLAM. In one embodiment, the apparatus obtains overlapping image frames and sensor inputs of an apparatus, wherein the sensor inputs comprise gyrometer data, accelerometer data and magnetometer data, splits computation work onto a plurality of vector processors to obtain six degree of freedom (6DoF) outputs of the apparatus based on a splitting algorithm, and performs a localization process to generate 6DoF estimations, and a mapping process to generate a cloud of three-dimensional points associated to the descriptors of the map. In one embodiment, the localization process and mapping process are configured to run sequentially. In another embodiment, the localization process and mapping process are configured to run in parallel.

What is claimed: 
     
         1 . An apparatus operating in an augmented reality/virtual reality (AR/VR) system comprising:
 a sensor unit that tracks series of motions of the apparatus and collects locale information of the apparatus relative to world coordinate system shared by a plurality of devices;   an initialization button that registers an initial six degree of freedom (6DoF) pose of the apparatus when pressed, wherein the initial 6DoF pose comprises a predefined three dimensions (3D) coordinate and an initial pose of the apparatus relative to the world coordinate system; and   one or more processors configured to communication with the AR/VR system and to generate series of 6DoF outputs of the apparatus based on the series of motion and locale information, wherein each 6DoF output comprises six dimensions of the apparatus including 3D of an orientation in a rotation space and three dimensions translation in a 3D space.   
     
     
         2 . The apparatus of  claim 1 , wherein the sensor unit comprises one or more cameras and an inertial measurement unit (IMU), and wherein the IMU detects movements, rotations, and magnetic headings of the apparatus. 
     
     
         3 . The apparatus of  claim 1 , wherein the sensor unit tracks motions of the apparatus and generates pose changes relative to the predefined coordinate relative to the world coordinate system. 
     
     
         4 . The apparatus of  claim 1 , further comprising a map module that obtains map data to generate a base map in reference to the world coordinate system and register the initial pose to the base map in reference to the world coordinate system. 
     
     
         5 . The apparatus of  claim 4 , wherein the apparatus determines real-time 6DoF outputs based the map data and locale information relative to the initial pose. 
     
     
         6 . A system comprising:
 a head mount device (HMD) with an HMD sensor unit and one or more processors configured to communication with the system and to generate series of six degree of freedom (6DoF) outputs of the HMD relative to a world coordinate system shared by a plurality of devices;   a game controller with a controller sensor unit and one or more processors configured to communication with the system and to generate series of 6DoF outputs of the controller relative to the world coordinate system; and   a two dimensional (2D) 6DoF tag attached to a first device selecting from the HMD and the game controller, wherein a second device determines an initial relative pose to the first device when the 6DoF tag is in the field of view of the second device.   
     
     
         7 . The system of  claim 6 , wherein the HMD sensor unit comprises one or more HMD cameras and an HMD inertial measurement unit (IMU) and the controller sensor unit comprises one or more controller cameras and a controller IMU. 
     
     
         8 . The system of  claim 6 , wherein the 6DoF tag is an AprilTag. 
     
     
         9 . The system of  claim 6 , wherein the controller sensor unit tracks motions of the apparatus and generates relative pose changes relative to the initial relative pose. 
     
     
         10 . The system of  claim 6 , wherein the one or more processors of the HMD device is further configured to split computation work onto a plurality of vector processors to 6DoF outputs based on a splitting algorithm. 
     
     
         11 . A method for an augmented reality/virtual reality (AR/VR) system, comprising:
 collecting locale information of an apparatus, wherein series of six degree of freedom (6DoF) outputs of the apparatus is generated based on locale information, and wherein each 6DoF output comprising six dimensions of the apparatus including three dimensions of an orientation in a rotation space and three dimensions translated in a 3D space;   obtaining a map in reference to a world coordinate system of the AR/VR system, wherein the map in reference to the world coordinate system is generated based on locale information collected by the apparatus;   initializing an initial 6DoF position of the apparatus relative to the map in reference to the world coordinate system by an initializing process selecting from pressing a physical button on an apparatus or capturing a two-dimensional (2D) 6DoF tag; and   performing a localization process to generate localization information of the apparatus based on the apparatus initial position.   
     
     
         12 . The method of  claim 11 , wherein the initializing is performed by pressing the physical button on the apparatus, and wherein the initializing registers the apparatus to a predefined coordinate on the map in reference to the world coordinate system. 
     
     
         13 . The method of  claim 11 , wherein the initializing is performed by capturing a 2D 6DoF tag, and wherein the 2D 6DoF tag is attached to a second apparatus. 
     
     
         14 . The method of  claim 13 , wherein the apparatus determines an initial relative pose to the second apparatus when the 2D 6DoF tag is in the field of view of the apparatus. 
     
     
         15 . The method of  claim 11 , further comprising: receiving real-time map data generated by the AR/VR system. 
     
     
         16 . The method of  claim 15 , wherein the apparatus determines 6DoF outputs based the received real-time map data. 
     
     
         17 . The method of  claim 11 , further comprising: splitting computation work onto a plurality of vector processors to obtain six degree of freedom (6DoF) outputs of the apparatus based on a splitting algorithm. 
     
     
         18 . The method of  claim 17 , wherein the splitting algorithm involves: dividing a current frame in N equal part; and each of a set of selected vector processors processes a portion of the current frame based on a split-by-corner rule, and wherein the split-by-corner rule determining whether each pixel of is a corner and classifying each pixel determined to a corner to a compressed descriptor by converting each sub-image centered by the pixel to a 16-float descriptor using a base matrix. 
     
     
         19 . The method of  claim 11 , wherein the localization process and mapping process are configured to run sequentially, wherein the localization process is split over all of the vector processors and the mapping process is split over all the vector processors. 
     
     
         20 . The method of  claim 11 , wherein the localization process and mapping process are configured to run in parallel, wherein the localization process is split over a first subset of the vector processors and the mapping process is split over the rest subset of the vector processors.