Systems and methods for augmented reality preparation, processing, and application

Abstract

Various of the disclosed embodiments provide systems and methods for acquiring and applying a depth determination of an environment in e.g., various augmented reality applications. A user may passively or actively scan a device (e.g., a tablet device, a mobile phone device, etc.) about the environment acquiring depth data for various regions. The system may integrate these scans into an internal three-dimensional model. This model may then be used in conjunction with subsequent data acquisitions to determine a device's location and orientation within the environment with high fidelity. In some embodiments, these determinations may be accomplished in real-time or near-real-time. Using the high-fidelity orientation and position determination, various augmented reality applications may then be possible using the same device used to acquire the depth data or a new device.

We claim: 
     
         1 . A device to facilitate an augmented reality (AR) experience, the device comprising:
 a depth sensor;   at least one processor;   at least one memory comprising instructions configured to cause the at least one processor to perform a method comprising:
 receiving depth information, the depth information comprising distances from the depth sensor to surfaces in a real-world environment; 
 determining a final pose, wherein determining the final pose comprises:
 orienting the depth information using a first pose; 
 determining correspondences between the depth information oriented using the first pose and at least a portion of a virtual model of the real-world environment; and 
 determining a final pose based upon the correspondences; and 
 
 making the final pose available to an AR application. 
   
     
     
         2 . The device of  claim 1 , wherein the first pose is a predicted pose comprising a belief probability distribution determined based, at least in part, upon the depth information. 
     
     
         3 . The device of  claim 1 , further comprising an inertial measurement unit (IMU), and wherein the predicted pose is further determined based upon IMU data acquired in conjunction with the acquisition of the depth information. 
     
     
         4 . The device of  claim 1 , wherein determining a final pose comprises iteratively orienting the depth information with a new pose determined based upon correspondences identified using a previous pose from a previous iteration, the final pose the last new pose so iteratively determined. 
     
     
         5 . The device of  claim 4 , wherein the iterations cease when successively determined poses resemble one another within a threshold, the threshold measured in meters-and-radians (MARs). 
     
     
         6 . The device of  claim 1 , wherein the virtual model comprises a Likelihood Field-Float (LFF) model or a Likelihood Field-Integer (LFI) model. 
     
     
         7 . The device of  claim 1 , wherein determining a final pose based upon the correspondences comprises applying a Pose Search Algorithm (PSA). 
     
     
         8 . The device of  claim 7 , wherein the PSA comprises a Scaling Series algorithm. 
     
     
         9 . The device of  claim 8 , further comprising an inertial measurement unit (IMU), wherein the first pose is a predicted pose, the predicted pose based upon inertial measurement unit (IMU) data acquired in conjunction with the acquisition of the depth information. 
     
     
         10 . The device of  claim 9 , wherein the device is a tablet. 
     
     
         11 . A device to determine a final pose, the device comprising:
 an interface configured to receive data from a second augmented reality (AR) experience device;   at least one processor;   at least one memory comprising instructions configured to cause the at least one processor to perform a method comprising:
 receiving depth information from the second device via the interface, the depth information comprising distances from a depth sensor to surfaces in a real-world environment; 
 determining a final pose, wherein determining the final pose comprises:
 orienting the depth information using a first pose; 
 determining correspondences between the depth information oriented using the first pose and at least a portion of a virtual model of the real-world environment; and 
 determining a final pose based upon the correspondences; and 
 
 making the final pose available to the second device. 
   
     
     
         12 . The device of  claim 11 , wherein the first pose is a predicted pose comprising a belief probability distribution determined based, at least in part, upon the depth information. 
     
     
         13 . The device of  claim 11 , wherein determining a final pose comprises iteratively orienting the depth information with a new pose determined based upon correspondences identified using a previous pose from a previous iteration, the final pose the last new pose so iteratively determined. 
     
     
         14 . The device of  claim 13 , wherein the iterations cease when successively determined poses resemble one another within a threshold, the threshold measured in meters-and-radians (MARs). 
     
     
         15 . The device of  claim 11 , wherein determining a final pose based upon the correspondences comprises applying a Pose Search Algorithm (PSA). 
     
     
         16 . The device of  claim 15 , wherein the PSA comprises a Scaling Series algorithm. 
     
     
         17 . A computer-implemented method for determining a pose of a device relative to a real-world environment, comprising:
 receiving depth information associated with the device, the depth information comprising distances from the device to surfaces in a real-world environment; and   determining a final pose, wherein determining the final pose comprises applying a Pose Search Algorithm (PSA) to a virtual model of the real-world environment and a first pose determined from the depth information.   
     
     
         18 . The computer-implemented method of  claim 17 , wherein the first pose is a predicted pose comprising a belief probability distribution determined based, at least in part, upon the depth information and upon inertial measurement unit (IMU) data acquired in conjunction with the acquisition of the depth information. 
     
     
         19 . The computer-implemented method of  claim 17 , wherein the PSA comprises a Scaling Series algorithm.