Three Dimensional User Interface Effects On A Display

Abstract

The techniques disclosed herein may use various sensors to infer a frame of reference for a hand-held device. In fact, with various inertial clues from accelerometer, pyrometer, and other instruments that report their states in real time, it is possible to track a Frenet frame of the device in real time to provide an instantaneous (or continuous) 3D frame-of-reference. In addition to—or in place of—calculating this instantaneous (or continuous) frame of reference, the position of a user's head may either be inferred or calculated directly by using one or more of a device's optical sensors, e.g., an optical camera, infrared camera, laser, etc. With knowledge of the 3D frame-of-reference for the display and/or knowledge of the position of the user's head, more realistic virtual 3D depictions of the graphical objects on the device's display may be created—and interacted with—by the user.

What claimed is: 
     
         1 . A graphical user interface method, comprising:
 receiving positional data from one or more position sensors disposed within a device;   determining a 3D frame of reference for the device based at least in part on the received positional data;   displaying a plurality of graphical user interface objects on a display of the device at a first orientation with respect to the 3D frame of reference;   receiving optical data from one or more optical sensors disposed within the device;   receiving non-optical data from one or more non-optical sensors;   determining a position of a user of the device's head based, at least in part, on the received optical data and the received non-optical data;   determining a position of a graphical layer with respect to at least one graphical user interface object based at least in part on the determined 3D frame of reference and the determined position of the user of the device's head, the graphical layer creating a lighting effect with respect to the at least one graphical user interface object;   generating a virtual 3D depiction of the at least one graphical user interface object on the display of the device with the graphical layer positioned with respect to the at least one graphical user interface object;   monitoring the position of the device and the position of the user of the device's head; and   adjusting the generated virtual 3D depiction of the at least one graphical user interface object on the display of the device, in response to movement of the device or the position of the user of the device's head, to depict at least one of: a top surface of the at least one graphical user interface object, a side surface of the at least one graphical user interface object, or behind the at least one graphical user interface object,   wherein the at least one graphical user interface object is represented in a virtual 3D operating system environment.   
     
     
         2 . The graphical user interface method of  claim 1 , wherein the act of generating further comprises applying an appropriate perspective transformation to the virtual 3D depiction of the at least one graphic& user interface object on the display of the device, wherein the transformation is based, at least in part, on the determined 3D frame of reference, the optical data, and the non-optical data. 
     
     
         3 . The graphical user interface method of  claim 1 , further comprising the act of receiving data from a touchscreen interface of the device, wherein the data received from the touchscreen interface is indicative of one or more locations in the virtual 3D operating system environment with which the user desires to interact. 
     
     
         4 . The graphical user interface method of  claim 1 , wherein the act of adjusting the generated virtual 3D depiction of the at least one graphical user interface object further comprises applying an appropriate perspective transformation to the virtual 3D depiction of the at least one graphical user interface object on the display of the device, wherein the transformation is based at least in part on a current determined 3D frame of reference for the device and the position of the user of the device's head. 
     
     
         5 . The graphical user interface method of  claim 1 , wherein the act of generating further comprises generating a virtual light source for the virtual 3D operating system environment. 
     
     
         6 . The graphical user interface method of  claim 1 , wherein the act of generating further comprises: ray tracing from the user's eyes through the display of the device and into the virtual 3D operating system environment. 
     
     
         7 . The graphical user interface method of  claim 1 , wherein the one or more non-optical sensors comprises one or more of: a sensor worn on the user of the device's head; a sensor on headgear worn by the user of the device; and a sensor that is part of a head-mounted display worn by the user of the device. 
     
     
         8 . A graphical user interface, comprising:
 a viewing surface;   a virtual 3D operating system environment;   one or more graphical user interface objects; and   a graphical layer positioned with respect to the one or more graphical user interface objects, the graphical layer creating a lighting effect with respect to the at least one graphical user interface object;   wherein the one or more graphical user interface objects and the graphical layer are represented in the virtual 3D operating system environment and depicted on the viewing surface, and   wherein the depiction of the one or more graphical user interface objects on the viewing surface and the graphical layer positioned with respect to the one or more graphical user interface objects is determined at least in part by a determined 3D frame of reference of the viewing surface with respect to a position of a head of a user of the viewing surface.   
     
     
         9 . The graphical user interface of  claim 8 , wherein the viewing surface comprises a touchscreen interface. 
     
     
         10 . The graphical user interface of  claim 9 , wherein data received from the touchscreen interface is indicative of one or more locations in the virtual 3D operating system environment with which the user of the viewing surface desires to interact. 
     
     
         11 . The graphical user interface of  claim 10 , further comprising one or more visual indicators on the viewing surface, wherein the one or more visual indicators are indicative of the one or more locations in the virtual 3D operating system environment with which the user of the viewing surface desires to interact. 
     
     
         12 . The graphical user interface of  claim 11 , wherein the locations of the one or more visual indicators on the viewing surface are corrected for parallax problems, wherein a determination of the corrected locations is based at least in part on the determined 3D frame of reference of the viewing surface. 
     
     
         13 . The graphical user interface of  claim 8 , wherein the depiction of the at least one graphical user interface object on the viewing surface comprises the application of one or more of the following visual effects to the at least one graphical user interface object: shading, shine, blur, and alpha masking. 
     
     
         14 . The graphical user interface of  claim 8 , wherein the position of the head of the user of the viewing surface is determined based, at least in part, on information received from one or more of: a sensor worn on the user's head; a sensor on headgear worn by the user; and a sensor that is part of a head-mounted display worn by the user. 
     
     
         15 . An apparatus, comprising:
 a display;   one or more optical sensors;   one or more positional sensors;   a memory; and   one or more programmable control devices communicatively coupled to the display, the one or more optical sensors, the one or more positional sensors, and the memory,   wherein the memory includes instructions for causing the one or more programmable control devices to:
 receive positional data from the one or more positional sensors; 
 determine a 3D frame of reference for the apparatus based at least in part on the received positional data; 
 display a plurality of graphical user interface objects on the display at a first orientation with respect to the 3D frame of reference; 
 receive optical data from the one or more optical sensors; 
 receive non-optical data from one or more non-optical sensors; 
 determine a position of a user's head based, at least in part, on the optical data and the non-optical data; 
   determine a position of a graphical layer with respect to the at least one graphical user interface object based at least in part on the determined 3D frame of reference and the determined position of the user's head, the graphical layer creating a lighting effect with respect to the at least one graphical user interface object;
 render a virtual 3D depiction of at least one graphical user interface object and the graphical layer positioned with respect to the at least one graphical user interface object on the display; 
 monitor the position of the apparatus and the position of the user's head; and 
 reposition the graphical layer with respect to the at least one graphical user interface object in response to a change in the 3D frame of reference based on the positional data or the position of the user's head to depict at least one of: 
   a top surface of the at least one graphical user interface object, a side surface of the at least one graphical user interface object, or behind the at least one graphical user interface object;
 wherein the at least one graphical user interface object is represented in a virtual 3D operating system environment. 
   
     
     
         16 . The apparatus of  claim 15 , wherein the one or more positional sensors comprise one or more of the following: a compass, an accelerometer, a GPS module, and a gyrometer. 
     
     
         17 . The apparatus of  claim 15 , wherein the one or more optical sensors comprise one or more of the following: a proximity sensor, an image sensor, and a video camera. 
     
     
         18 . The apparatus of  claim 15 , wherein the one or more non-optical sensors comprises one or more of: a sensor worn on the user's head; a sensor on headgear worn by the user; and a sensor that is part of a head-mounted display worn by the user. 
     
     
         19 . The apparatus of  claim 15 , wherein the instructions to render further comprise instructions to apply an appropriate perspective transformation to the virtual 3D depiction of the at least one graphical user interface object. 
     
     
         20 . The apparatus of  claim 19 , wherein the perspective transformation is based, at least in part, on the determined 3D frame of reference, the optical data, and the non-optical data.