Speech Recognition Using an Operating System Hooking Component for Context-Aware Recognition Models

Abstract

Inputs provided into user interface elements of an application are observed. Records are made of the inputs and the state(s) the application was in while the inputs were provided. For each state, a corresponding language model is trained based on the input(s) provided to the application while the application was in that state. When the application is next observed to be in a previously-observed state, a language model associated with the application's current state is applied to recognize speech input provided by a user and thereby to generate speech recognition output that is provided to the application. An application's state at a particular time may include the user interface element(s) that are displayed and/or in focus at that time, and is determined by an operating system hooking component embedded in the automatic speech recognition system.

What is claimed is: 
     
         1 . A computer-implemented method performed by at least one computer processor executing computer program instructions tangibly stored on at least one non-transitory computer-readable medium, the computer-implemented method comprising using the at least one computer processor to perform the operations of:
 receiving, by an automatic speech recognition system executed by the at least one computer processor, a plurality of inputs into an application while the application is in a first state;   observing, by the automatic speech recognition system, a frequency at which each input of the plurality of inputs is received into the application;   tailoring, by the automatic speech recognition system, a first language model associated with the application, based on the plurality of inputs, wherein tailoring the first language model comprises:   generating the first language model by assigning, in the first language model, a probability to each input of the plurality of inputs based on the observed frequency;   determining, by the automatic speech recognition system, that the application is in the first state; and   applying, by the automatic speech recognition system, the tailored first language model to a first speech input in response to determining that the application is in the first state.   
     
     
         2 . The computer-implemented method of  claim 1 , wherein a probability assigned to a particular input of the plurality of inputs is equal to the observed frequency at which the particular input is received into the application. 
     
     
         3 . The computer-implemented method of  claim 1 , wherein a probability assigned to a particular input is adjusted upward in response to observing that the particular input has been received into the application. 
     
     
         4 . The computer-implemented method of  claim 1 , wherein a probability assigned to a particular input is adjusted downward in response to observing that the particular input has not been received into the application. 
     
     
         5 . The computer implemented method of  claim 1 , wherein determining that the application is in the first state is based on information provided by a context sharing application. 
     
     
         6 . The computer-implemented method of  claim 1 , wherein determining that the application is in the first state comprises:
 analyzing application data to determine that the application is in the first state;   comparing the determined first state of the application to a state associated with the tailored first language model; and   determining that the first state of the application and the state associated with the tailored first language model are same states.   
     
     
         7 . The computer-implemented method of  claim 1 , wherein determining that the application is in the first state comprises:
 comparing application data of the application to application data associated with the tailored first language model; and   determining that the application data of the application and the application data associated with the tailored first language model are same data.   
     
     
         8 . The computer-implemented method of  claim 1 , wherein applying the tailored first language model to the first speech input comprises applying the tailored first language model to the first speech input after achieving a degree of confidence in a level of accuracy of the tailored first language model. 
     
     
         9 . The computer-implemented method of  claim 1 , wherein applying the tailored first language model to the first speech input comprises applying the tailored first language model to the first speech input when a number of the plurality of inputs associated with the tailored first language model exceeds a predefined threshold. 
     
     
         10 . The computer-implemented method of  claim 1 , further comprising receiving application data and identifications of a state of the application, wherein the plurality of inputs includes text, speech, and mixed text and speech inputs. 
     
     
         11 . A non-transitory computer readable medium storing computer program instructions which, when executed by at least one computer processor, cause the at least one computer processor to:
 receive, by an automatic speech recognition system executed by the at least one computer processor, a plurality of inputs into an application while the application is in a first state;   observe, by the automatic speech recognition system, a frequency at which each input of the plurality of inputs is received into the application;   tailor, by the automatic speech recognition system, a first language model associated with the application, based on the plurality of inputs, wherein to tailor the first language model, the computer program instructions cause the at least one computer processor to:   generate the first language model by assigning, in the first language model, a probability to each input of the plurality of inputs based on the observed frequency;   determine, by the automatic speech recognition system, that the application is in the first state; and   apply, by the automatic speech recognition system, the tailored first language model to a first speech input in response to the determination that the application is in the first state.   
     
     
         12 . The non-transitory computer readable medium of  claim 11 , wherein a probability assigned to a particular input of the plurality of inputs is equal to the observed frequency at which the particular input is received into the application. 
     
     
         13 . The non-transitory computer readable medium of  claim 11 , wherein a probability assigned to a particular input is adjusted upward in response to an observation that the particular input has been received into the application. 
     
     
         14 . The non-transitory computer readable medium of  claim 11 , wherein a probability assigned to a particular input is adjusted downward in response to an observation that the particular input has not been received into the application. 
     
     
         15 . The non-transitory computer readable medium of  claim 11 , wherein the computer program instructions, when executed by the at least one computer processor, cause the at least one computer processor to determine that the application is in the first state based on information provided by a context sharing application. 
     
     
         16 . The non-transitory computer readable medium of  claim 11 , wherein to determine that the application is in the first state, the computer program instructions, when executed by the at least one computer processor, cause the at least one computer processor to:
 analyze application data to determine that the application is in the first state;   compare the determined first state of the application to a state associated with the tailored first language model; and   determine that the first state of the application and the state associated with the tailored first language model are same states.   
     
     
         17 . The non-transitory computer readable medium of  claim 11 , wherein to determine that the application is in the first state, the computer program instructions, when executed by the at least one computer processor, cause the at least one computer processor to:
 compare application data of the application to application data associated with the tailored first language model; and   determine that the application data of the application and the application data associated with the tailored first language model are same data.   
     
     
         18 . The non-transitory computer readable medium of  claim 11 , wherein the computer program instructions, when executed by the at least one computer processor, cause the at least one computer processor to apply the tailored first language model to the first speech input after achieving a degree of confidence in a level of accuracy of the tailored first language model. 
     
     
         19 . The non-transitory computer readable medium of  claim 11 , wherein the computer program instructions, when executed by the at least one computer processor, cause the at least one computer processor to apply the tailored first language model to the first speech input when a number of the plurality of inputs associated with the tailored first language model exceeds a predefined threshold. 
     
     
         20 . An automated speech recognition system comprising:
 means for receiving a plurality of inputs into an application while the application is in a first state;   means for observing a frequency at which each input of the plurality of inputs is received into the application;   means for tailoring a first language model associated with the application, based on the plurality of inputs, wherein the means for tailoring the first language model generates the first language model by assigning, in the first language model, a probability to each input of the plurality of inputs based on the observed frequency;   means for determining that the application is in the first state; and   means for applying the tailored first language model to a first speech input in response to determining that the application is in the first state.