Virtual object display interface between a wearable device and a mobile device

Abstract

Systems, devices, media, and methods are presented for displaying a virtual object on the display of a portable eyewear device using motion data gathered by a face-tracking application on a mobile device. A controller engine leverages the processing power of a mobile device to locate the face supporting the eyewear, locate the hand holding the mobile device, acquire the motion data, and calculate an apparent path of the virtual object. The virtual object is displayed in a series of locations along the apparent path, based on both the course traveled by the mobile device (in the hand) and the track traveled by the eyewear device (on the face), so that the virtual object is persistently viewable to the user.

What is claimed is: 
     
       1. A method of displaying a virtual object, comprising:
 capturing frames of video data with a mobile camera coupled to a mobile device in motion along a course relative to a three-dimensional coordinate system; 
 estimating a wearable device position relative to the three-dimensional coordinate system, the wearable device comprising a display characterized by a field of view; 
 calculating a virtual path based on the course traversed by the mobile device, wherein at least a portion of the virtual path is located outside the field of view; and 
 presenting on the display a virtual object along the virtual path as an overlay relative to the mobile device and in accordance with the wearable device position. 
 
     
     
       2. The method of  claim 1 , wherein the display is semi-transparent and facilitates a view of a physical environment, and wherein the presenting further comprises:
 overlaying the virtual object relative to the view of the physical environment, such that the virtual object is persistently viewable relative to the physical environment. 
 
     
     
       3. The method of  claim 1 , wherein the calculating further comprises:
 defining a track traveled by the wearable device based on the wearable device position over time; and 
 calculating the virtual path based on the track, such that the virtual path is correlated with both the course of the mobile device and the track of the wearable device. 
 
     
     
       4. The method of  claim 3 , further comprising:
 detecting a face in the captured frames of video data, wherein the face supports the wearable device; 
 estimating a face location relative to the three-dimensional coordinate system; and 
 defining the track in accordance with the face location. 
 
     
     
       5. The method of  claim 4 , further comprising:
 acquiring one or more anchors associated with the face; 
 estimating the face location based on the one or more anchors; and 
 selectively suspending the detecting such that a subsequent face is not detected. 
 
     
     
       6. The method of  claim 1 , further comprising:
 detecting a hand in the captured frames of video data, wherein the hand supports the mobile device; 
 estimating a hand location relative to the three-dimensional coordinate system; and 
 presenting the virtual object in accordance with the hand location. 
 
     
     
       7. The method of  claim 1 , wherein the display comprises a projector and a semi-transparent lens assembly, and wherein the presenting further comprises:
 projecting the virtual object onto the lens assembly. 
 
     
     
       8. The method of  claim 1 , further comprising:
 selecting the virtual object from a collection of objects associated with an experience application. 
 
     
     
       9. A system for displaying a virtual object comprising:
 a wearable device comprising a processor, a memory, and a display characterized by a field of view; 
 a mobile device comprising a mobile camera; 
 programming in the memory, wherein execution of the programming by the processor configures the wearable device to perform functions, including functions to: 
 capture frames of video data with the mobile camera, wherein the mobile device is in motion along a course relative to a three-dimensional coordinate system; 
 estimate a wearable device position relative to the three-dimensional coordinate system; 
 calculate a virtual path based on the course traversed by the mobile device, wherein at least a portion of the virtual path is located outside the field of view; and 
 present on the display a virtual object along the virtual path as an overlay relative to the mobile device and in accordance with the wearable device position. 
 
     
     
       10. The system of  claim 9 , wherein the display is semi-transparent and facilitates a view of a physical environment, and wherein the execution further configures the wearable device to perform additional functions, including functions to:
 overlay the virtual object relative to the view of the physical environment, such that the virtual object is persistently viewable relative to the physical environment. 
 
     
     
       11. The system of  claim 9 , wherein the execution further configures the wearable device to perform additional functions, including functions to:
 define a track traveled by the wearable device based on the wearable device position over time; and 
 calculate the virtual path based on the track, such that the virtual path is correlated with both the course of the mobile device and the track of the wearable device. 
 
     
     
       12. The system of  claim 11 , wherein the execution further configures the wearable device to perform additional functions, including functions to:
 detect a face in the captured frames of video data, wherein the face supports the wearable device; 
 estimate a face location relative to the three-dimensional coordinate system; and 
 define the track in accordance with the face location. 
 
     
     
       13. The system of  claim 12 , wherein the execution further configures the wearable device to perform additional functions, including functions to:
 acquire one or more anchors associated with the face; 
 estimate the face location based on the one or more anchors; and 
 selectively suspend the detecting such that a subsequent face is not detected. 
 
     
     
       14. The system of  claim 9 , wherein the execution further configures the wearable device to perform additional functions, including functions to:
 detect a hand in the captured frames of video data, wherein the hand supports the mobile device; 
 estimate a hand location relative to the three-dimensional coordinate system; and 
 present the virtual object in accordance with the hand location. 
 
     
     
       15. The system of  claim 9 , wherein the display comprises a projector and a semi-transparent lens assembly, and wherein the execution further configures the wearable device to perform additional functions, including functions to:
 select the virtual object from a collection of objects associated with an experience application; and 
 project the virtual object onto the lens assembly. 
 
     
     
       16. A non-transitory computer-readable medium storing program code which, when executed, is operative to cause an electronic processor to perform the steps of:
 capturing frames of video data with a mobile camera coupled to a mobile device in motion along a course relative to a three-dimensional coordinate system; 
 estimating a wearable device position relative to the three-dimensional coordinate system, the wearable device comprising the electronic processor and a semi-transparent display characterized by a field of view; 
 calculating a virtual path based on the course traversed by the mobile device, wherein at least a portion of the virtual path is located outside the field of view; and 
 presenting on the display a virtual object along the virtual path as an overlay relative to the mobile device and in accordance with the wearable device position. 
 
     
     
       17. The non-transitory computer-readable medium of  claim 16 , wherein the program code when executed is operative to cause an electronic processor to perform the further steps of:
 defining a track traveled by the wearable device based on the wearable device position over time; 
 calculating the virtual path based on the track, such that the virtual path is correlated with both the course of the mobile device and the track of the wearable device; 
 detecting a face in the captured frames of video data, wherein the face supports the wearable device; 
 estimating a face location relative to the three-dimensional coordinate system; and 
 defining the track in accordance with the face location. 
 
     
     
       18. The non-transitory computer-readable medium of  claim 17 , wherein the program code when executed is operative to cause an electronic processor to perform the further steps of:
 acquiring one or more anchors associated with the face; 
 estimating the face location based on the one or more anchors; and 
 selectively suspending the detecting such that a subsequent face is not detected. 
 
     
     
       19. The non-transitory computer-readable medium of  claim 16 , wherein the program code when executed is operative to cause an electronic processor to perform the further steps of:
 detecting a hand in the captured frames of video data, wherein the hand supports the mobile device; 
 estimating a hand location relative to the three-dimensional coordinate system; and 
 presenting the virtual object in accordance with the hand location. 
 
     
     
       20. The non-transitory computer-readable medium of  claim 16 , wherein the display comprises a projector and a semi-transparent lens assembly, and wherein the program code when executed is operative to cause an electronic processor to perform the further steps of:
 selecting the virtual object from a collection of objects associated with an experience application; and 
 projecting the virtual object onto the lens assembly.