Multimodal text input system, such as for use with touch screens on mobile phones

Abstract

A system and method for entering text from a user includes a programmed processor that receives inputs from the user and disambiguates the inputs to present word choices corresponding to the text. In one embodiment, inputs are received in two or more modalities and are analyzed to present the word choices. In another embodiment, a keyboard is divided into zones each of which represents two more input characters. A sequence of zones selected by the user is analyzed to present word choices corresponding to the zone selected.

I/We claim: 
     
         1 . A computer system, comprising:
 one or more processors;   a memory configured to store programmed instructions;   a first input device;   a second input device different from the first input device; and   processor electronics configured to execute the programmed instructions to perform operations of receiving text inputs from a user by:
 receiving a first input in a first modality through the first input device; 
 receiving a second input, through the second input device, in a second modality different from the first modality,
 wherein a combination of the first input in the first modality and the second input in the second modality are indicative of text to be entered; 
 
 analyzing the combination of the first input in the first modality and the second input in the second modality to select one or more word choices; and 
 using the selected word choices to determine the text to be entered. 
   
     
     
         2 . The computer system of  claim 1 ,
 wherein the first input device is a keyboard;   wherein the second input device is a microphone;   wherein receiving the first input in the first modality comprises receiving text input via the keyboard;   wherein receiving the second input in the second modality comprises receiving audio input via the microphone; and   wherein analyzing the combination of the first input in the first modality and the second input in the second modality to determine the one or more word choices corresponding to the text to be entered is performed by:
 using the text input to determine multiple potential word choices; 
 presenting the multiple potential word choices to the user; and 
 using the audio input to select one of the multiple potential word choices. 
   
     
     
         3 . The computer system of  claim 1 ,
 wherein the first input device is a keyboard;   wherein the second input device is a microphone;   wherein receiving the first input in the first modality comprises receiving audio input via the microphone;   wherein receiving the second input in the second modality comprises receiving text input via the keyboard; and   wherein analyzing the combination of the first input in the first modality and the second input in the second modality to determine the one or more word choices corresponding to the text to be entered is performed by:
 using the audio input to determine multiple potential word choices; 
 presenting the multiple potential word choices to the user; and 
 using the text input to select one of the multiple potential word choices. 
   
     
     
         4 . The computer system of  claim 1 , wherein analyzing the combination of the first input in the first modality and the second input in the second modality to determine the one or more word choices corresponding to the text to be entered is performed by:
 using the first input in the first modality to determine multiple potential word choices;   presenting the multiple potential word choices to the user; and   using the second input in the second modality to select one of the multiple potential word choices.   
     
     
         5 . The computer system of  claim 1 ,
 wherein the first input device is a keyboard;   wherein the second input device is a microphone;   wherein receiving the first input in the first modality comprises receiving input via the keyboard; and   wherein receiving the second input in the second modality comprises receiving input via the microphone.   
     
     
         6 . The computer system of  claim 1 ,
 wherein the first input device is a tactile input;   wherein the second input device is a microphone;   wherein receiving the first input in the first modality comprises receiving input via the tactile input; and   wherein receiving the second input in the second modality comprises receiving input via the microphone.   
     
     
         7 . The computer system of  claim 1 ,
 wherein the operations of receiving text inputs from the user are further performed by:
 determining recent actions taken by the user; and 
 selecting the one or more word choices based on a determined correspondence between the one or more word choices and the recent actions taken by the user. 
   
     
     
         8 . The computer system of  claim 1 ,
 wherein the operations of receiving text inputs from the user are further performed by:
 determining a location of the user; and 
 selecting the one or more word choices based on a determined correspondence between the one or more word choices and the location of the user. 
   
     
     
         9 . The computer system of  claim 1 ,
 wherein the first input device is a virtual keyboard;   wherein the second input device is a microphone;   wherein receiving the first input in the first modality comprises receiving a swipe across the virtual keyboard;   wherein receiving the second input in the second modality comprises receiving a sequence of one or more taps to the virtual keyboard;   wherein the operations of receiving text inputs from the user are further performed by receiving a third input in a third modality via the microphone; and   wherein the combination of the first input in the first modality and the second input in the second modality is further combined with the third input in the third modality to collectively determine the one or more word choices corresponding to the text to be entered.   
     
     
         10 . The computer system of  claim 1 ,
 wherein the first input device is an accelerometer; and   wherein receiving the first input in the first modality comprises receiving input via the accelerometer.   
     
     
         11 . The computer system of  claim 1 ,
 wherein the operations of receiving text inputs from the user further include, after receiving the first input in a first modality, determining an ambiguity in the first input; and   wherein the second input is received in response to determining the ambiguity in the first input.   
     
     
         12 . A computer-readable storage medium, excluding a transitory signal, storing instructions that, when executed by a computing system, cause the computing system to perform operations for receiving text inputs from a user, the operations comprising:
 receiving a first input in a first modality;   receiving a second input in a second modality different from the first modality,
 wherein a combination of the first input in the first modality and the second input in the second modality are indicative of text to be entered; 
   analyzing the combination of the first input in the first modality and the second input in the second modality to select one or more word choices; and   using the selected word choices to determine the text to be entered.   
     
     
         13 . The computer-readable storage medium of  claim 12 ,
 wherein the computing system includes a keyboard and a microphone;   wherein receiving the first input in the first modality comprises receiving audio input via the microphone;   wherein receiving the second input in the second modality comprises receiving text input via the keyboard; and   wherein analyzing the combination of the first input in the first modality and the second input in the second modality to determine the one or more word choices corresponding to the text to be entered is performed by:
 using the audio input to determine multiple potential word choices; 
 presenting the multiple potential word choices to the user; and 
 using the text input to select one of the multiple potential word choices. 
   
     
     
         14 . The computer-readable storage medium of  claim 12 ,
 wherein the operations for receiving text inputs from the user further include, after receiving the first input in a first modality, determining an ambiguity in the first input; and   wherein the second input is received in response to determining the ambiguity in the first input.   
     
     
         15 . The computer-readable storage medium of  claim 12 ,
 wherein the operations for receiving text inputs from the user are further performed by:
 determining recent actions taken by the user; and 
 selecting the one or more word choices based on a determined correspondence between the one or more word choices and the recent actions taken by the user. 
   
     
     
         16 . The computer-readable storage medium of  claim 12 ,
 wherein the computing system includes a virtual keyboard and a microphone;   wherein receiving the first input in the first modality comprises receiving a swipe across the virtual keyboard;   wherein receiving the second input in the second modality comprises receiving a sequence of one or more taps to the virtual keyboard;   wherein the operations for receiving text inputs from the user are further performed by receiving a third input in a third modality via the microphone; and   wherein the combination of the first input in the first modality and the second input in the second modality is further combined with the third input in the third modality to collectively select the one or more word choices.   
     
     
         17 . A method for receiving text inputs from a user to a computing system, the method comprising:
 receiving a first input in a first modality;   receiving a second input in a second modality different from the first modality,
 wherein a combination of the first input in the first modality and the second input in the second modality are indicative of text to be entered; 
   analyzing the combination of the first input in the first modality and the second input in the second modality to select one or more word choices; and   using the selected word choices to determine the text to be entered.   
     
     
         18 . The method of  claim 17 ,
 wherein the computing system includes a keyboard and a microphone;   wherein receiving the first input in the first modality comprises receiving audio input via the microphone;   wherein receiving the second input in the second modality comprises receiving text input via the keyboard; and   wherein analyzing the combination of the first input in the first modality and the second input in the second modality to determine the one or more word choices corresponding to the text to be entered is performed by:
 using the audio input to determine multiple potential word choices; 
 presenting the multiple potential word choices to the user; and 
 using the text input to select one of the multiple potential word choices. 
   
     
     
         19 . The method of  claim 17  further comprising:
 after receiving the first input in a first modality, determining an ambiguity in the first input; 
 wherein the second input is received in response to determining the ambiguity in the first input. 
 
     
     
         20 . The method of  claim 17 ,
 wherein the computing system includes a virtual keyboard and a microphone;   wherein receiving the first input in the first modality comprises receiving a swipe across the virtual keyboard;   wherein receiving the second input in the second modality comprises receiving a sequence of one or more taps to the virtual keyboard;   wherein the method further comprises receiving a third input in a third modality via the microphone; and   wherein the combination of the first input in the first modality and the second input in the second modality is further combined with the third input in the third modality, which is used to select the one or more word choices.