Artificial reality system having a sliding menu

Abstract

An artificial reality system is described that renders, presents, and controls user interface elements within an artificial reality environment, and performs actions in response to one or more detected gestures of the user. The artificial reality system can include a menu that can be activated and interacted with using one hand. In response to detecting a menu activation gesture performed using one hand, the artificial reality system can cause a menu to be rendered. A menu sliding gesture (e.g., horizontal motion) of the hand can be used to cause a slidably engageable user interface (UI) element to move along a horizontal dimension of the menu while horizontal positioning of the UI menu is held constant. Motion of the hand orthogonal to the menu sliding gesture (e.g., non-horizontal motion) can cause the menu to be repositioned. The implementation of the artificial reality system does require use of both hands or use of other input devices in order to interact with the artificial reality system.

What is claimed is: 
     
       1. An artificial reality system comprising:
 an image capture device configured to capture image data; 
 a head mounted display (HMD) configured to output artificial reality content;
 a gesture detector comprising processing circuitry configured to identify, from the image data, a menu activation gesture comprising a configuration of a hand in a substantially upturned orientation of the hand and a pinching configuration of a thumb and a finger of the hand and identify, from the image data and subsequent to the menu activation gesture, a menu sliding gesture comprising the configuration of the hand in combination with a motion of the hand; 
 a user interface (UI) engine configured to, in response to the menu activation gesture, generate a menu interface and a slidably engageable UI element at a first position relative to the menu interface, and in response to the menu sliding gesture that, via at least the configuration of the hand, engages the slidably engageable UI element, translate the slidably engageable UI element to a second position relative to the menu interface by sliding the slidably engageable UI element such that each motion of the hand, in at least one dimension, is translated to a motion of the menu sliding gesture that causes the sliding of the slidably engageable UI element; and 
 a rendering engine configured to render the artificial reality content, the menu interface, and the translation of the slidably engageable UI element from the first position relative to the user menu interface to the second position relative to the user menu interface for display at the HMD. 
 
 
     
     
       2. The artificial reality system of  claim 1 ,
 wherein the menu interface comprises one or more menu items arrayed along a dimension of the menu interface, and 
 wherein the UI engine is configured to highlight one of the menu items according to a position of the slidably engageable UI element relative to the menu interface. 
 
     
     
       3. The artificial reality system of  claim 2 , wherein the one or more menu items correspond to respective applications executing on the artificial reality system. 
     
     
       4. The artificial reality system of  claim 1 ,
 wherein the menu interface comprises one or more menu items arrayed along a dimension of the menu interface, 
 wherein to translate the slidably engageable UI element to the second position relative to the menu interface, the UI engine is configured to slide the slidably engageable UI element along the dimension of the menu interface to the second position relative to the menu interface. 
 
     
     
       5. The artificial reality system of  claim 1 ,
 wherein the menu sliding gesture comprises motion of the hand in a substantially first direction, 
 wherein the gesture detector is configured to identify, from the image data, motion of the hand in a substantially second direction subsequent to the menu activation gesture, the substantially second direction being substantially orthogonal to the substantially first direction, 
 wherein the UI engine is further configured to, in response to the motion of the hand in the substantially second direction, translate the slidably engageable UI element and the menu interface while retaining a position of the slidably engageable UI element relative to the menu interface. 
 
     
     
       6. The artificial reality system of  claim 1 , further comprising:
 an application engine comprising processing circuitry for execution of one or more artificial reality applications, 
 wherein the gesture detector is configured to identify, from the image data, a selection gesture subsequent to the menu sliding gesture, and 
 wherein the application engine is configured to perform an action in response to the selection gesture. 
 
     
     
       7. The artificial reality system of  claim 6 , wherein the selection gesture comprises one of (1) movement of a different finger of the hand, (2) translation of the hand in a direction that is substantially normal to the menu interface, or (3) reconfiguring the thumb and the finger of the hand to no longer be in the pinching configuration. 
     
     
       8. The artificial reality system of  claim 1 ,
 wherein the gesture detector is further configured to identify, from the image data, a menu prompt gesture prior to the menu activation gesture, and 
 wherein the UI engine is further configured to generate a menu prompt element in response to the menu prompt gesture. 
 
     
     
       9. The artificial reality system of  claim 8 , wherein the menu prompt gesture comprises the hand configured in a substantially upturned position with a space between a thumb and a finger, and
 wherein the UI engine generates the menu prompt element in the space between a thumb and a finger of a virtual hand. 
 
     
     
       10. The artificial reality system of  claim 9 , wherein the menu prompt element comprises a line between the thumb and the finger of the virtual hand. 
     
     
       11. The artificial reality system of  claim 1 , wherein the image capture device is integrated with the HMD. 
     
     
       12. A method comprising:
 obtaining, by an artificial reality system including a head mounted display (HMD), image data via an image capture device; 
 identifying, by the artificial reality system from the image data, a menu activation gesture, the menu activation gesture comprising a configuration of a hand in a substantially upturned orientation of the hand and a pinching configuration of a thumb and a finger of the hand; 
 generating, by the artificial reality system in response to the menu activation gesture, a menu interface and a slidably engageable UI element at a first position relative to the menu interface; 
 identifying, from the image data and subsequent to the menu activation gesture, a menu sliding gesture comprising the configuration of the hand in combination with a motion of the hand; 
 in response to the menu sliding gesture that, via at least the configuration of the hand, engages the slidably engageable UI element, translate the slidably engageable UI element to a second position relative to the menu interface, by sliding the slidably engageable UI element such that each motion of the hand, in at least one dimension, is translated to a motion of the menu sliding gesture that causes the sliding of the slidably engageable UI element; and 
 rendering, by the artificial reality system, artificial reality content, the menu interface, and the translation of the slidably engageable UI element from the first position relative to the menu interface to the second position relative to the user menu interface for display at the HMD. 
 
     
     
       13. The method of  claim 12 , wherein the menu interface comprises one or more menu items arrayed along a dimension of the menu interface, the method further comprising:
 highlighting, by the artificial reality system, one of the menu items according to a position of the slidably engageable UI element relative to the menu interface. 
 
     
     
       14. The method of  claim 12 , wherein translating the slidably engageable UI element to the second position relative to the menu interface comprises sliding the slidably engageable UI element along a dimension of the menu interface to the second position relative to the menu interface. 
     
     
       15. The method of  claim 12 ,
 wherein the menu sliding gesture comprises motion of the hand in a substantially first direction, the method further comprising:
 identifying, by the artificial reality system from the image data, motion of the hand in a substantially second direction subsequent to the menu activation gesture, the substantially second direction being substantially orthogonal to the substantially first direction; and 
 translating, by the artificial reality system in response to the motion of the hand in the substantially second direction, the slidably engageable UI element and the menu interface while retaining a position of the slidably engageable UI element relative to the menu interface. 
 
 
     
     
       16. The method of  claim 12 , further comprising:
 identifying, by the artificial reality system from the image data, a menu prompt gesture; and 
 generating, by the artificial reality system, a menu prompt element in response to the menu prompt gesture. 
 
     
     
       17. A non-transitory, computer-readable medium comprising instructions that, when executed, cause one or more processors of an artificial reality system to:
 capture image data via an image capture device; 
 identify, from the image data, a menu activation gesture comprising a configuration of the a hand; 
 in response to the menu activation gesture, generate a menu interface and a slidably engageable UI element at a first position relative to the menu interface; 
 identify, subsequent to the menu activation gesture, a menu sliding gesture comprising the configuration of the hand in combination with a motion of the hand; 
 in response to the menu sliding gesture that, via at least the configuration of the hand, engages the slidably engageable UI element, translate the slidably engageable UI element to a second position relative to the menu interface by sliding the slidably engageable UI element such that each motion of the hand, in at least one dimension, is translated to a motion of the menu sliding gesture that causes the sliding of the slidably engageable UI element; and 
 render artificial reality content, the menu interface, and the slidably engageable UI element for display at a head mounted display (HMD). 
 
     
     
       18. The non-transitory, computer-readable medium of  claim 17 , wherein the instructions further cause the one or more processors to:
 identify, from the image data, a menu prompt gesture comprising the hand configured in a substantially upturned position with a space between a thumb and a finger; and 
 generate, in the space between a thumb and a finger of a virtual hand, a menu prompt element in response to the menu prompt gesture. 
 
     
     
       19. The non-transitory, computer-readable medium of  claim 18 , wherein the menu prompt element comprises a line between the thumb and the finger of the virtual hand.  
     
     
       20. The non-transitory, computer-readable medium of  claim 17 , wherein the menu interface comprises one or more menu items, and wherein the instructions further cause the one or more processors to:
 modify the appearance of one of the menu items according to a position of the slidably engageable UI element relative to the menu interface.    
     
     
       21. The non-transitory, computer-readable medium of  claim 20 , wherein the one or more menu items correspond to respective applications executing on the artificial reality system.  
     
     
       22. The non-transitory, computer-readable medium of  claim 17 ,
 wherein the menu interface comprises one or more menu items arrayed along a dimension of the menu interface, and   wherein the slidably engageable UI element is rendered translating along the dimension of the menu interface from the first position relative to the menu interface to the second position relative to the menu interface.    
     
     
       23. The non-transitory, computer-readable medium of  claim 22 , wherein the one or more menu items correspond to respective applications executing on the artificial reality system.  
     
     
       24. The non-transitory, computer-readable medium of  claim 17 , wherein the menu sliding gesture comprises motion of the hand in a substantially first direction, and wherein the instructions further cause the one or more processors to:
 identify, subsequent to the menu activation gesture, motion of the hand in a substantially second direction, the substantially second direction being substantially orthogonal to the substantially first direction; and   translate, in response to the motion of the hand in the substantially second direction, the slidably engageable UI element and the menu interface while retaining a position of the slidably engageable UI element relative to the menu interface.    
     
     
       25. The non-transitory, computer-readable medium of  claim 17 , wherein the instructions further cause the one or more processors to:
 identify, subsequent to the menu sliding gesture, a selection gesture; and   perform an action in response to the selection gesture.    
     
     
       26. The non-transitory, computer-readable medium of  claim 25 , wherein the menu interface comprises one or more menu items, the second position relates to a first menu item of the one or more menu items, and the performed action is associated with the first menu item.  
     
     
       27. The non-transitory, computer-readable medium of  claim 25 ,
 wherein the configuration of the hand of the menu activation gesture comprises the hand in a substantially upturned orientation and a pinching configuration of a thumb and a finger of the hand; and   wherein the selection gesture comprises one of (1) movement of a different finger of the hand, (2) translation of the hand in a direction that is substantially normal to the menu interface, or (3) reconfiguring the thumb and the finger of the hand to not be in a pinching configuration.    
     
     
       28. The non-transitory, computer-readable medium of  claim 17 , wherein the instructions further cause the one or more processors to:
 render a virtual hand for display at the HMD.    
     
     
       29. The non-transitory, computer-readable medium of  claim 28 , wherein the virtual hand comprises one or more digits.  
     
     
       30. A method comprising:
 capturing image data via an image capture device;   identifying, from the image data, a menu activation gesture comprising a configuration of a hand;   in response to the menu activation gesture, generating a menu interface and a slidably engageable UI element at a first position relative to the menu interface;   identifying, subsequent to the menu activation gesture, a menu sliding gesture comprising the configuration of the hand in combination with a motion of the hand; and   in response to the menu sliding gesture that, via at least the configuration of the hand, engages the slidably engageable UI element, translating the slidably engageable UI element to a second position relative to the menu interface, and rendering the menu interface and the slidably engageable UI element for display at a head mounted display (HMD),
 wherein the slidably engageable UI element is translated to the second position by sliding the slidably engageable UI element such that each motion of the hand, in at least one dimension, is translated to a motion of the menu sliding gesture that causes the sliding of the slidably engageable UI element.  
   
     
     
       31. The method of  claim 30 , further comprising:
 identifying a menu prompt gesture; and   generating a menu prompt element in response to the menu prompt gesture.    
     
     
       32. The method of  claim 31 , wherein the menu prompt gesture comprises the hand configured in a substantially upturned position with a space between a thumb and a finger.  
     
     
       33. The method of  claim 31 , wherein the menu prompt element is generated in the space between a thumb and a finger of a virtual hand.  
     
     
       34. The method of  claim 33 , wherein the menu prompt element comprises a line between the thumb and the finger of the virtual hand.  
     
     
       35. The method of  claim 30 , wherein the menu interface comprises one or more menu items, the method further comprising:
 modifying the appearance of one of the menu items according to a position of the slidably engageable UI element relative to the menu interface.    
     
     
       36. The method of  claim 35 , wherein the one or more menu items correspond to respective applications executing on an artificial reality system.  
     
     
       37. The method of  claim 30 ,
 wherein the menu interface comprises one or more menu items arrayed along a dimension of the menu interface, and   wherein translating the slidably engageable UI element to the second position relative to the menu interface comprises translating the slidably engageable UI element along the dimension of the menu interface from the first position relative to the menu interface to the second position relative to the menu interface.    
     
     
       38. The method of  claim 37 , wherein the one or more menu items correspond to respective applications executing on an artificial reality system.  
     
     
       39. The method of  claim 30 , wherein the menu sliding gesture comprises motion of the hand in a substantially first direction, the method further comprising:
 identifying, subsequent to the menu activation gesture, motion of the hand in a substantially second direction, the substantially second direction being substantially orthogonal to the substantially first direction; and   translating, in response to the motion of the hand in the substantially second direction, the slidably engageable UI element and the menu interface while retaining a position of the slidably engageable UI element relative to the menu interface.    
     
     
       40. The method of  claim 30 , further comprising:
 identifying, subsequent to the menu sliding gesture, a selection gesture; and   performing an action in response to the selection gesture.    
     
     
       41. The method of  claim 40 , wherein the menu interface comprises one or more menu items, the second position relates to a first menu item of the one or more menu items, and the performed action is associated with the first menu item.  
     
     
       42. The method of  claim 40 ,
 wherein the configuration of the hand of the menu activation gesture comprises the hand in a substantially upturned orientation and a pinching configuration of a thumb and a finger of the hand; and   wherein the selection gesture comprises one of (1) movement of a different finger of the hand, (2) translation of the hand in a direction that is substantially normal to the menu interface, or (3) reconfiguring a thumb and a finger of the hand to not be in a pinching configuration.    
     
     
       43. The method of  claim 30 , further comprising:
 rendering a virtual hand for display at the HMD.    
     
     
       44. The method of  claim 43 , wherein the virtual hand comprises one or more digits.  
     
     
       45. An artificial reality system comprising:
 an image capture device configured to capture image data;   a head mounted display (HMD) configured to output artificial reality content;   a gesture detector comprising processing circuitry configured to identify, from the image data, a menu activation gesture comprising a configuration of a hand and identify, subsequent to the menu activation gesture, a menu sliding gesture comprising the configuration of the hand in combination with a motion of the hand,   a user interface (UI) engine comprising processing circuitry configured to, in response to the menu activation gesture, generate a menu interface and a slidably engageable UI element at a first position relative to the menu interface, and in response to the menu sliding gesture that, via at least the configuration of the hand, engages the slidably engageable UI element, translate the slidably engageable UI element to a second position relative to the menu interface by sliding the slidably engageable UI element such that each motion of the hand, in at least one dimension, is translated to a motion of the menu sliding gesture that causes the sliding of the slidably engageable UI element; and   a rendering engine comprising processing circuitry configured to render the menu interface and the slidably engageable UI element for display at the HMD.    
     
     
       46. The artificial reality system of  claim 45 ,
 wherein the gesture detector is further configured to identify a menu prompt gesture, and   wherein the UI engine is further configured to generate a menu prompt element in response to the menu prompt gesture.    
     
     
       47. The artificial reality system of  claim 46 , wherein the menu prompt gesture comprises the hand configured in a substantially upturned position with a space between a thumb and a finger.  
     
     
       48. The artificial reality system of  claim 46 , wherein the UI engine generates the menu prompt element in the space between a thumb and a finger of a virtual hand.  
     
     
       49. The artificial reality system of  claim 48 , wherein the menu prompt element comprises a line between the thumb and the finger of the virtual hand.  
     
     
       50. The artificial reality system of  claim 45 ,
 wherein the menu interface comprises one or more menu items, and   wherein the UI engine is configured to modify the appearance of one of the menu items according to a position of the slidably engageable UI element relative to the menu interface.    
     
     
       51. The artificial reality system of  claim 50 , wherein the one or more menu items correspond to respective applications executing on the artificial reality system.  
     
     
       52. The artificial reality system of  claim 45 ,
 wherein the menu interface comprises one or more menu items arrayed along a dimension of the menu interface,   wherein to translate the slidably engageable UI element to the second position relative to the menu interface, the UI engine is configured to translate the slidably engageable UI element along the dimension of the menu interface from the first position relative to the menu interface to the second position relative to the menu interface.    
     
     
       53. The artificial reality system of  claim 52 , wherein the one or more menu items correspond to respective applications executing on the artificial reality system.  
     
     
       54. The artificial reality system of  claim 45 ,
 wherein the menu sliding gesture comprises motion of the hand in a substantially first direction,   wherein the gesture detector is configured to identify motion of the hand in a substantially second direction subsequent to the menu activation gesture, the substantially second direction being substantially orthogonal to the substantially first direction, and   wherein the UI engine is further configured to, in response to the motion of the hand in the substantially second direction, translate the slidably engageable UI element and the menu interface while retaining a position of the slidably engageable UI element relative to the menu interface.    
     
     
       55. The artificial reality system of  claim 45 , further comprising:
 an application engine comprising processing circuitry for execution of one or more artificial reality applications,
 wherein the gesture detector is configured to identify a selection gesture subsequent to the menu sliding gesture, and 
 wherein the application engine is configured to perform an action in response to the selection gesture.  
   
     
     
       56. The artificial reality system of  claim 55 , wherein the menu interface comprises one or more menu items, the second position relates to a first menu item of the one or more menu items, and the performed action is associated with the first menu item.  
     
     
       57. The artificial reality system of  claim 55 ,
 wherein the configuration of the hand of the menu activation gesture comprises the hand in a substantially upturned orientation and a pinching configuration of a thumb and a finger of the hand; and   wherein the selection gesture comprises one of (1) movement of a different finger of the hand, (2) translation of the hand in a direction that is substantially normal to the menu interface, or (3) reconfiguring the thumb and the finger of the hand to not be in a pinching configuration.    
     
     
       58. The artificial reality system of  claim 45 , wherein the image capture device is integrated with the HMD.  
     
     
       59. The artificial reality system of  claim 45 , wherein the rendering engine is further configured to render a virtual hand for display at the HMD.  
     
     
       60. The artificial reality system of  claim 59 , wherein the virtual hand comprises one or more digits.