Portable electronic device for mixed reality headset

Abstract

A portable electronic device is used with a mixed reality (MR) headset. The MR headset includes a lens and a holder that retains the portable electronic device in a defined orientation relative to the lens. The portable electronic device includes a camera, a display device, and a processor. The camera outputs video frames and is arranged to view the lens. The display device is arranged to display information that is projected on the lens for reflection directly or indirectly toward the user's eyes and the camera. The processor performs operations that include processing the video frames from the camera to identify locations of at least one real-world feature, displaying information on the display device, and controlling at least one of the processing of the video frames and the displaying of the information on the display device to at least partially reduce occurrence in the video frames of the displayed information.

1 . A portable electronic device for a mixed reality (MR) headset including a lens through which a user wearing the MR headset can view real-world features, the portable electronic device comprising:
 a camera configured to output video frames and arranged to view at least a portion of the lens of the MR headset;   a display device arranged to display information that is to be projected on the lens for reflection directly or indirectly toward the user's eyes and the camera;   at least one processor operationally connected to the camera and to the display device, and configured to perform operations comprising:
 processing the video frames from the camera to identify a location of a real-world feature captured in the video frames, 
 wherein the identification of the location of the real-world feature is based on the video frames that were captured while the display device ceases to display information for projection on the lens and is not based on the video frames that were captured while the display device displays information for projection on the lens. 
   
     
     
         2 . The portable electronic device of  claim 1 , wherein the operations cease the processing, to identify the location of the real-world feature, of at least some of the video frames based on being captured by the camera while the information is displayed on the display for projection on the lens, and the operations resume the processing, to identify the location of the real-world feature, of the video frames based on being captured by the camera while the information is not being displayed on the display. 
     
     
         3 . The portable electronic device of  claim 2 , wherein the operations to resume the processing of the video frames based on being captured by the camera while the information is not being displayed on the display, comprise skipping without processing to identify location of the real-world feature in at least some of the video frames that were captured by the camera while the information is being displayed on the display for projection on the lens. 
     
     
         4 . The portable electronic device of  claim 3 , wherein the operations to skip without processing to identify location of the real-world feature in at least some of the video frames that were captured by the camera while the information is being displayed on the display for projection on the lens, comprise skipping all of the video frames that were captured by the camera while the information is being displayed on the display for projection on the lens. 
     
     
         5 . The portable electronic device of  claim 1 , wherein the operations repetitively alternate between displaying the information on the display device and then ceasing the display of the information while initiating the identification of the location of the real-world feature in the video frames from the camera which are captured by the camera while the display of the information is ceased, wherein the the identification of the location of the real-world feature is controlled so at least some of the video frames captured by the camera while the information is displayed on the display device are not used to identify the location of the real-world feature. 
     
     
         6 . The portable electronic device of  claim 5 , wherein all the video frames that were captured by the camera while the information is displayed on the display device are skipped without being used to identify the location of the real-world feature. 
     
     
         7 . The portable electronic device of  claim 5 , wherein the operations further comprise:
 determining when a display quality prioritization rule is satisfied; and   responsive to the display quality prioritization rule becoming satisfied, increasing a duty cycle of a first time duration during which the display device is repetitively refreshed to display the information relative to a second time duration during which the location of the real-world feature is determined based on the video frames from the camera which are captured by the camera while the display of the information is ceased.   
     
     
         8 . The portable electronic device of  claim 5 , wherein the operations further comprise:
 determining when a camera video prioritization rule is satisfied; and   responsive to the camera video prioritization rule becoming satisfied, decreasing a duty cycle of a first time duration during which the display device is repetitively refreshed to display the information relative to a second time duration during which the location of the real-world feature is determined based on the video frames from the camera which are captured by the camera while the display of the information is ceased.   
     
     
         9 . The portable electronic device of  claim 1 , wherein the operations further comprise:
 determining when a display quality prioritization rule is satisfied; and   responsive to the display quality prioritization rule becoming satisfied, increasing a frequency at which the display device is repetitively refreshed to display the information and/or decreasing a frequency at which the location of the real-world feature is determined based on the video frames from the camera which are captured by the camera while the display of the information is ceased.   
     
     
         10 . The portable electronic device of  claim 1 , wherein the operations further comprise:
 determining when a camera video prioritization rule is satisfied; and   responsive to the camera video prioritization rule becoming satisfied, decreasing a frequency at which the display device is repetitively refreshed to display the information and/or increasing a frequency at which the location of the real-world feature is determined based on the video frames from the camera which are captured by the camera while the display of the information is ceased.   
     
     
         11 . The portable electronic device of  claim 1 , wherein the operations further comprise:
 determining an amount of overlap occurring between: 1) a region of the video frames from the camera that captures the information being displayed by the display device; and 2) location of the real-world feature in the video frames; and   based on the amount of overlap satisfying a defined rule, initiate operation to not perform the identification of the location of the real-world feature on at least some of the video frames that were captured while the display device displays information for projection on the lens.   
     
     
         12 . The portable electronic device of  claim 1 , wherein the operations further comprise:
 displaying a calibration pattern on the display device;   processing at least one of the video frames to determine a region of the at least one of the video frames containing the calibration pattern;   ceasing display of the calibration pattern on the display device;   following cessation of display of the calibration pattern, processing the video frames subsequently captured by the camera to determine when location of the real-world feature enters the region of the video frames; and   responsive to the location of the real-world feature entering the region of the video frames, initiating the operations to not perform the identification of the location of the real-world feature on at least some of the video frames that were captured while the display device displays information for projection on the lens.   
     
     
         13 . The portable electronic device of  claim 1 , wherein:
 the operations further comprise processing the video frames from the camera to identify a first region of the video frames from the camera that captures the information being displayed by the display device; and   the operations to not perform the identification of the location of the real-world feature from being performed on at least some of the video frames that were captured while the display device displays information for projection on the lens, further comprise:
 processing the first region of the video frames at a lower frequency than a second region of the video frames, which is outside the first region of the video frames and does not capture the information being displayed by the display device, to identify locations of the real-world feature which can be in the first region and/or the second region. 
   
     
     
         14 . The portable electronic device of  claim 1 , wherein the operations further comprise:
 processing the video frames from the camera to identify a region of the display device that displays a portion of the information and which at least partially overlaps a location of the real-world feature in the video frames; and   decreasing brightness and/or contrast of the region of the display device.   
     
     
         15 . The portable electronic device of  claim 1 , wherein the operations further comprise:
 processing the video frames from the camera to identify a region of the display device that displays a portion of the information and which at least partially overlaps a location of the real-world feature in the video frames; and   determining a primary color of the real-world feature and selecting a color of the information that is then displayed in the region of the display device to differentiate from the primary color of the real-world feature.   
     
     
         16 . The portable electronic device of  claim 1 , wherein the operations further comprise:
 processing the video frames from the camera to identify a first region of the display device that displays a portion of the information and which at least partially overlaps a location of the real-world feature in the video frames; and   increasing brightness and/or contrast of a second region of the display device that does not overlap the first region.   
     
     
         17 . The portable electronic device of  claim 1 , wherein the operations further comprise:
 processing the video frames from the camera to identify a first region of the display device that displays a portion of the information and which at least partially overlaps a location of the real-world feature in the video frames; and   determining a primary color of the real-world feature and selecting a color of the information that is then displayed in a second region of the display device that does not overlap the first region to differentiate from the primary color of the real-world feature.   
     
     
         18 . The portable electronic device of  claim 1 , wherein the portable electronic device is connectable to a holder of the MR headset that is configured to releasably retain the portable electronic device in a defined orientation relative to the lens of the MR headset, wherein the camera of the portable electronic device is arranged by the holder to view at least a portion of the lens of the MR headset. 
     
     
         19 . A method by a portable electronic device for a mixed reality (MR) headset including a lens through which a user wearing the MR headset can view real-world features, wherein a display device of the portable electronic device is arranged to display information that is projected on the lens for reflection directly or indirectly toward the user's eyes and a camera of the portable electronic device, the method comprising:
 process the video frames from the camera to identify a location of a real-world feature captured in the video frames, wherein the identification of the location of the real-world feature is based on the video frames that were captured while the display device ceases to display information for projection on the lens and is not based on the video frames that were captured while the display device displays information for projection on the lens.   
     
     
         20 . A computer program product for a portable electronic device for a mixed reality (MR) headset having a lens through which a user wearing the MR headset can view real-world features, wherein a display device of the portable electronic device is arranged to display information that is projected on the lens for reflection directly or indirectly toward the user's eyes and a camera of the portable electronic device, the computer program product comprising:
 a non-transitory computer readable medium storing program code executable by at least one processor of the portable electronic device to perform operations to:
 process the video frames from the camera to identify a location of a real-world feature captured in the video frames, wherein the identification of the location of the real-world feature is based on the video frames that were captured while the display device ceases to display information for projection on the lens and is not based on the video frames that were captured while the display device displays information for projection on the lens.