Multimodal interface

Abstract

A system includes a processor and a multimodal graphical user interface. The multimodal graphical user interface causes the processor to display a menu of a plurality of selectable menu items. Each menu item has an associated command name and an associated unique hand shape that corresponds to the command name. The multimodal graphical user interface causes the processor to select a given menu item of the plurality of selectable menu items based at least in part on a combination of a cursor input and a selection gesture input.

What is claimed is: 
     
         1 . A system comprising:
 a processor; and   a multimodal graphical user interface to cause the processor to:
 display a menu of a plurality of selectable menu items, wherein each menu item has an associated command name and an associated unique hand shape corresponding to the command name; and 
 select a given menu item of the plurality of selectable menu items based at least in part on a combination of a cursor input and a selection gesture input. 
   
     
     
         2 . The system of  claim 1 , wherein the multimodal graphical interface causes the processor to select the given menu item based at least in part on a speech input. 
     
     
         3 . The system of  claim 2 , wherein the speech input corresponds to the command name associated with the given menu item. 
     
     
         4 . The system of  claim 1 , wherein the multimodal graphical user interface causes the processor to select the given menu item in response to a hand shape gesture input. 
     
     
         5 . The system of  claim 4 , wherein the hand shape gesture input corresponds to the unique hand shape associated with the given menu. 
     
     
         6 . The system of  claim 1 , wherein the given menu item of the plurality of selectable menu items includes at least one sub-menu item, wherein the at least one sub-menu item is associated with another command name and another unique hand shape, and wherein the multimodal graphical user interface causes the processor to select a sub-menu item of the at least one sub-menu item in response to a combination of the cursor input and the selection gesture input. 
     
     
         7 . The system of  claim 6 , wherein the multimodal graphical user interface causes the processor to select a given sub-menu of the at least one sub-menu item in response to a speech input. 
     
     
         8 . The system of  claim 1 , wherein the multimodal graphical user interface causes the processor to display the plurality of selectable menu items in at least one of the following layouts: radial, vertical and horizontal. 
     
     
         9 . A method comprising:
 displaying, with a display device, a display associated with a multimodal graphical user interface, wherein the display comprises a menu of a plurality of selectable menu items, and wherein the displaying comprises displaying an associated command name and an associated unique hand shape corresponding to the command name for each menu item of the plurality of selectable menu items;   detecting, by a computing device, a combination of a cursor input and a selection gesture input and associating the detected combination with a given menu item of the plurality of selectable menu items; and   performing, by the computing device, a function associated with the given menu item.   
     
     
         10 . The method of  claim 9 , further comprising detecting by the computing device, the given menu item in response to a hand shape gesture input. 
     
     
         11 . The method of  claim 10 , wherein the hand shape gesture input corresponds to the unique hand shape associated with the given menu item. 
     
     
         12 . The system of  claim 1 , wherein displaying comprises displaying the plurality of selectable menu items in at least one of the following layouts: radial, vertical and horizontal. 
     
     
         13 . An article comprising a non-transitory computer readable storage medium storing instructions that when executed by a processor-based system cause the processor-based system to:
 display a menu of a plurality of selectable menu items;   detect a cursor input;   detect a selection gesture input; and   in response to a combination of the cursor input and the hand selection gesture representing selection of a given menu item of the plurality of selectable menu items, perform a function associated with the given menu item.   
     
     
         14 . The article of  claim 13 , the storage medium storing instructions that when executed by the processor-based system cause the processor-based system to select a sub-menu item associated with the given menu item. 
     
     
         15 . The article of  claim 14 , the storage medium storing instructions that when executed by the processor-based system cause the processor-based system to select the sub-menu based at least in part on a speech input or a hand shape gesture input.