Automatically determining language for speech recognition of spoken utterance received via an automated assistant interface

Abstract

Determining a language for speech recognition of a spoken utterance received via an automated assistant interface for interacting with an automated assistant. Implementations can enable multilingual interaction with the automated assistant, without necessitating a user explicitly designate a language to be utilized for each interaction. Implementations determine a user profile that corresponds to audio data that captures a spoken utterance, and utilize language(s), and optionally corresponding probabilities, assigned to the user profile in determining a language for speech recognition of the spoken utterance. Some implementations select only a subset of languages, assigned to the user profile, to utilize in speech recognition of a given spoken utterance of the user. Some implementations perform speech recognition in each of multiple languages assigned to the user profile, and utilize criteria to select only one of the speech recognitions as appropriate for generating and providing content that is responsive to the spoken utterance.

We claim: 
     
       1. A method implemented by one or more processors, the method comprising:
 determining that a user has interacted with one or more applications when the one or more applications were providing natural language content in a first language, wherein the first language is different from a second language that is a user-specific speech processing language defined in user-specific language profile that corresponds to the user; 
 causing, based on determining that the user has interacted with the one or more applications, the user-specific language profile to be modified to also reference the first language; 
 receiving, subsequent to the user-specific language profile being modified to also reference the first language, audio data corresponding to a spoken utterance received at an automated assistant interface of a computing device; 
 responsive to receiving the audio data corresponding to the first spoken utterance, and based on the first language being included in the user-specific language profile and the second language being the user-specific speech processing language in the user-specific language profile, determining:
 a first score that characterizes a probability that the spoken utterance was provided by the user in the first language, and 
 a second score that characterizes another probability that the spoken utterance was provided by the user in the second language; 
 
 selecting, based on at least the first score and the second score, the first language over the second language; and 
 in response to selecting the first language, using first language speech recognition results, from performing speech recognition for the first language based on the audio data, in generating a response to the spoken utterance. 
 
     
     
       2. The method of  claim 1 , wherein determining the first score is based on output that is generated in performing speech recognition for the first language based on the audio data. 
     
     
       3. The method of  claim 2 , wherein determining the second score is based on output that is generated in performing speech recognition for the second language based on the audio data. 
     
     
       4. The method of  claim 3 , wherein determining the first score is further based on a first probability metric that is assigned, in the user-specific language profile, to the first language. 
     
     
       5. The method of  claim 4 , wherein determining the first score is further based on a second probability metric that is assigned, in the user-specific language profile, to the second language. 
     
     
       6. The method of  claim 1 , wherein determining the first score is based on a first probability metric that is assigned, in the user-specific language profile, to the first language. 
     
     
       7. The method of  claim 6 , further comprising utilizing the first probability, in determining the first score, in response to the first probability being associated with a location and the computing device having the location when the audio data, corresponding to the spoken utterance, is received. 
     
     
       8. The method of  claim 6 , wherein determining the first score is further based on a second probability metric that is assigned, in the user-specific language profile, to the second language. 
     
     
       9. The method of  claim 1 , further comprising:
 causing the response to be rendered at the computing device, wherein the response is provided in the first language in response to selecting the first language. 
 
     
     
       10. The method of  claim 1 , wherein the one or more applications include an email application and wherein determining that the user has interacted with the one or more application when the one or more application were providing natural language content in the first language comprises determining that the email application includes emails written in the first language. 
     
     
       11. A system comprising:
 one or more processors; and 
 memory configured to store instructions that, when executed by the one or more processors cause the one or more processors to perform operations that include: 
 determining that a user has interacted with natural language content that is in a first language, wherein the first language is different from a second language that is a user-specific speech processing language defined in user-specific language profile that corresponds to the user; 
 causing, based on determining that the user has interacted with the natural language content in the first language, the user-specific language profile to be modified to also reference the first language; 
 receiving, subsequent to the user-specific language profile being modified to also reference the first language, audio data corresponding to a spoken utterance received at an automated assistant interface of a computing device; 
 responsive to receiving the audio data corresponding to the first spoken utterance, and based on the first language being included in the user-specific language profile and the second language being the user-specific speech processing language in the user-specific language profile, determining:
 a first score that characterizes a probability that the spoken utterance was provided by the user in the first language, and 
 a second score that characterizes another probability that the spoken utterance was provided by the user in the second language; 
 
 selecting, based on at least the first score and the second score, the first language over the second language; and 
 in response to selecting the first language, using first language speech recognition results, from performing speech recognition for the first language based on the audio data, in generating a response to the spoken utterance. 
 
     
     
       12. The system of  claim 11 , wherein determining the first score is based on output that is generated in performing speech recognition for the first language based on the audio data. 
     
     
       13. The system of  claim 12 , wherein determining the second score is based on output that is generated in performing speech recognition for the second language based on the audio data. 
     
     
       14. The system of  claim 13 , wherein determining the first score is further based on a first probability metric that is assigned, in the user-specific language profile, to the first language. 
     
     
       15. The system of  claim 14 , wherein determining the first score is further based on a second probability metric that is assigned, in the user-specific language profile, to the second language. 
     
     
       16. The system of  claim 11 , wherein determining the first score is based on a first probability metric that is assigned, in the user-specific language profile, to the first language. 
     
     
       17. The system of  claim 16 , wherein the operations further include utilizing the first probability, in determining the first score, in response to the first probability being associated with a location and the computing device having the location when the audio data, corresponding to the spoken utterance, is received. 
     
     
       18. The system of  claim 16 , wherein determining the first score is further based on a second probability metric that is assigned, in the user-specific language profile, to the second language. 
     
     
       19. The system of  claim 11 , wherein the operations further include:
 causing the response to be rendered at the computing device, wherein the response is provided in the first language in response to selecting the first language. 
 
     
     
       20. The system of  claim 11 , wherein the one or more applications include an email application and wherein determining that the user has interacted with the one or more application when the one or more application were providing natural language content in the first language comprises determining that the email application includes emails written in the first language.