Compound gesture-speech commands

Abstract

A multimedia entertainment system combines both gestures and voice commands to provide an enhanced control scheme. A user's body position or motion may be recognized as a gesture, and may be used to provide context to recognize user generated sounds, such as speech input. Likewise, speech input may be recognized as a voice command, and may be used to provide context to recognize a body position or motion as a gesture. Weights may be assigned to the inputs to facilitate processing. When a gesture is recognized, a limited set of voice commands associated with the recognized gesture are loaded for use. Further, additional sets of voice commands may be structured in a hierarchical manner such that speaking a voice command from one set of voice commands leads to the system loading a next set of voice commands.

1 . A method for controlling a computing system using a voice commands, comprising:
 accessing multiple depth images from a depth sensor system;   recognizing a gesture from the multiple depth images;   in response to recognizing a gesture, choosing a subset of a set of sound commands based on the recognized gesture, the set of sound commands includes multiple subsets, each subset is associated with one or more gestures and sound command recognition data for the respective subset;   receiving sound input;   recognizing a sound command from the chosen subset based on the sound input; and   performing an action in response to the recognized sound command.   
     
     
         2 . The method of  claim 1 , wherein:
 the depth images include a two-dimensional pixel area of a captured scene, the pixels in the two-dimensional pixel area represent depth values of one or more objects in the scene captured by the depth sensor system.   
     
     
         3 . The method of  claim 1 , further comprising:
 loading sound command recognition data for the chosen subset of sound commands.   
     
     
         4 . The method of  claim 1 , wherein:
 the sound input is received from a microphone.   
     
     
         5 . The method of  claim 1 , wherein:
 the recognizing includes attempting to match the sound input to sound commands in the chosen subset and not to sound commands in other subsets of the set of sound commands.   
     
     
         6 . The method of  claim 1 , further comprising:
 displaying the chosen subset of sound commands in response to the recognizing the gesture.   
     
     
         7 . The method of  claim 6 , wherein:
 the recognized gesture selects a displayed object; and   the chosen subset of sound commands is displayed proximate to the selected displayed object.   
     
     
         8 . The method of  claim 1 , further comprising:
 after performing the action in response to the recognized sound command, loading additional sound command recognition data for a related subset of the chosen subset of sound commands;   receiving a further sound command;   recognizing the further sound command using the loaded additional sound command recognition data; and   performing an action in response to the recognized further sound command.   
     
     
         9 . The method of  claim 1 , further comprising:
 changing the state of the computing system based on the recognized gesture; and   providing a hierarchical subset of sound commands related to the changed state of the computing system.   
     
     
         10 . The method of  claim 1 , wherein the depth images and the sound command are received substantially simultaneously. 
     
     
         11 . The method of  claim 1 , wherein:
 the depth images include a two-dimensional pixel area of a captured scene, each pixel in the two-dimensional pixel area represents a depth value of one or more objects in the scene captured by the depth sensor system;   the method further comprises displaying the chosen subset of sound commands in response to the recognizing the gesture;   the sound input is received after displaying the chosen subset of sound commands; and   the recognizing includes attempting to match the sound input to sound commands in the chosen subset and not to sound commands in other subsets of the set of sound commands.   
     
     
         12 . A computing system, comprising:
 a monitor for displaying multimedia content;   a depth sensor for capturing depth images;   a microphone for capturing sounds; and   a processor in communication with the depth sensor, the microphone and the monitor; the processor communicates with the monitor to display an object, the processor receives multiple depth images from the depth sensor and recognizes a gesture from the multiple depth images, the processor chooses a subset of a set of sound commands based on and in response to the recognized gesture, the set of sound commands includes multiple subsets, the processor loads sound command recognition data for the chosen subset of sound commands in response to the recognized gesture, the processor receives sound input from the microphone and recognizes a sound command from the chosen subset based on the sound input and the loaded sound command recognition data without searching all of the set of sound commands, processor performs an action in response to the recognized sound command.   
     
     
         13 . The system of  claim 12 , wherein:
 the subset of sound commands is organized in one or more hierarchical levels, wherein each hierarchical level is only loaded as necessary to process a gesture.   
     
     
         14 . The system of  claim 13 , wherein:
 a subsequent hierarchical level of sound commands is loaded and displayed only after a sound command from a previous hierarchical level has been recognized and the desired action associated with the sound command taken.   
     
     
         15 . The system of  claim 12 , wherein:
 the gesture is used by the processor to facilitate recognition of the sound command, and the sound command is used by the processor to facilitate recognition of the gesture.   
     
     
         16 . The system of  claim 12 , wherein:
 the processor recognizes the gesture by attempting to match the sound input to sound commands in the chosen subset and not to sound commands in other subsets of the set of sound commands.   
     
     
         17 . The system of  claim 12 , wherein:
 the processor displays the chosen subset of sound commands in response to the recognizing the gesture, and receives the sound input after displaying the chosen subset of sound commands.   
     
     
         18 . A processor readable storage device having instructions thereon, the instructions for programming one or more processors to perform a method, comprising:
 receiving sound input;   accessing multiple depth images from a depth sensor;   recognizing a gesture from the multiple depth images based on tracking motion of an object moving in a scene captured in the depth images;   in response to recognizing the gesture, choosing a subset of a set of sound commands based on the recognized gesture; and   identifying a sound command using a recognition process that is limited to attempting to match the sound input to sound commands of the chosen subset and not to sound commands of the set of sound commands not in the chosen subset.   
     
     
         19 . The processor readable storage device of  claim 18 , wherein:
 the method further comprises displaying the chosen subset of sound commands in response to the recognizing the gesture, the sound input is received after displaying the chosen subset of sound commands.   
     
     
         20 . The processor readable storage device of  claim 18 , wherein:
 the recognizing the gesture from the multiple depth images based on tracking motion of the object comprises performing skeletal tracking of the object based on the depth images and determining a gesture being performed based on the skeletal tracking.