Computing services using embeddings of a transformer-based encoder

Abstract

Techniques are described and relate to providing computing services using embeddings of a transformer-based encoder. In an example, a computer system generates, by using a machine learning (ML) transformer, an embedding vector based at least in part on text. The computer system stores the embedding vector and an association between the embedding vector and the text in a data store. Further, the computer system determines that a task is to be performed based at least in part on natural language understanding (NLU) of the text. The computer system receives the embedding vector from the data store based at least in part on the association between the embedding vector and the text. The task is performed based at least in part on the embedding vector after being received from the data store.

What is claimed is: 
     
       1. A method implemented on a computer system, the method comprising:
 receiving, from a user device, audio data that corresponds to a user utterance detected by the user device; 
 generating text by performing automatic speech recognition (ASR) on the audio data; 
 inputting the text to a machine learning (ML) transformer of an artificial intelligence (AI) model, the ML transformer trained to generate embeddings; 
 storing, in a data store, an embedding vector and an association between the embedding vector and at least a portion of the text, the embedding vector generated by the ML transformer for the at least portion of the text; 
 determining, by performing natural language understanding (NLU) on the text, that a task is to be performed based at least in part on the at least portion of the text, the task comprising one or more of: intent classification, named entity recognition, sentiment analysis, natural language input prediction, search continuity prediction, or ASR correction; 
 sending a query to the data store based at least in part on the task; 
 receiving, from the data store in response to the query, the embedding vector based at least in part on the association between the embedding vector and the at least portion of the text; 
 performing the task by inputting the embedding vector to a multi-task layer (MTL) of the AI model, the MTL trained to perform multiple tasks; and 
 sending, to the user device, data based at least in part on performance of the task. 
 
     
     
       2. The method of  claim 1 , further comprising:
 determining a word from the text, wherein the at least portion of the text comprises the word; 
 determining a position of the word in the text; and 
 storing, in the data store, metadata for the embedding vector, wherein the metadata indicates the association and comprises the position of the word and an identifier of the user utterance. 
 
     
     
       3. The method of  claim 2 , further comprising:
 generating the query based at least in part on a determination that performing the task depends on information associated with the position in the text, wherein the query indicates the position and the identifier of the user utterance, wherein the embedding vector is received based at least in part on the metadata associating the position with the embedding vector and on the identifier of the user utterance. 
 
     
     
       4. The method of  claim 1 , further comprising:
 storing, in the data store, a first embedding vector and a second embedding vector, wherein an output of the ML transformer comprises the first embedding vector and the second embedding vector, wherein the first embedding vector corresponds to a set of words from the text and is usable to perform a first task of the multiple tasks, and wherein the second embedding vector corresponds to a different set of words from the text and is usable to perform a second task of the multiple tasks. 
 
     
     
       5. A computer system comprising:
 one or more processors; and 
 one or more memories storing computer-readable instructions that, upon execution by the one or more processors, configure the computer system to:
 generate, by using a machine learning (ML) transformer, an embedding vector based at least in part on text, the embedding vector generated prior to performing natural language understanding (NLU) processing on the text; 
 store the embedding vector and an association between the embedding vector and the text in a data store; 
 determine that a task is to be performed based at least in part on the NLU processing of the text; 
 send, based at least in part on the task, a query to the data store; 
 receive, in response to the query, the embedding vector from the data store based at least in part on the association between the embedding vector and the text; and 
 perform the task based at least in part on the embedding vector after being received from the data store. 
 
 
     
     
       6. The computer system of  claim 5 , wherein the one or more memories store further computer-readable instructions that, upon execution by the one or more processors, additionally configure the computer system to:
 receive, from a user device, audio data that corresponds to a user utterance detected by the user device; 
 generate the text by at least performing automatic speech recognition (ASR) on the audio data; 
 input the text to the ML transformer, the ML transformer trained to generate embeddings; 
 input the embedding vector to a multi-task layer (MTL) of an AI model, the MTL trained to perform multiple tasks, wherein the AI model comprises the ML transformer; and 
 send, to the user device, data based at least in part on performance of the task. 
 
     
     
       7. The computer system of  claim 5 , wherein the embedding vector is a first embedding vector for a first set of words from the text, wherein the one or more memories store further computer-readable instructions that, upon execution by the one or more processors, additionally configure the computer system to:
 generate a second embedding vector for a second set of words from the text; and 
 store, in the data store, the second embedding vector, wherein the association stored in the data store is an association between the first embedding vector and at least one of: the first set of words or position information of the first set of words in the text. 
 
     
     
       8. The computer system of  claim 7 , wherein the one or more memories store further computer-readable instructions that, upon execution by the one or more processors, additionally configure the computer system to:
 determine that performing the task depends on information associated with the first set of words or the position information; and 
 send the query to the data store that indicates the first set of words or the position information, wherein the embedding vector is received in response to the query. 
 
     
     
       9. The computer system of  claim 5 , wherein the embedding vector is a first embedding vector for a first set of words from the text, wherein the task is a first task from multiple tasks of a multi-task layer (MTL) of an AI model that comprises the ML transformer, and wherein the one or more memories store further computer-readable instructions that, upon execution by the one or more processors, additionally configure the computer system to:
 generate, by using the ML transformer, a second embedding vector for a second set of words from the text, wherein the first embedding vector is usable to perform the first task, and wherein the second embedding vector is usable to perform a second task. 
 
     
     
       10. The computer system of  claim 9 , wherein the first set of words is different from the second set of words, and wherein the association stored in the data store associates the first embedding vector with the first set of words or with positions of words from the first set in the text. 
     
     
       11. The computer system of  claim 10 , wherein the one or more memories store further computer-readable instructions that, upon execution by the one or more processors, additionally configure the computer system to:
 determine that, to perform the first task, information corresponding to the positions is needed; and 
 generate the query that indicates the positions, wherein the first embedding vector instead of the second embedding vector is received in response of the query based at least in part on the positions. 
 
     
     
       12. The computer system of  claim 9 , wherein the first embedding vector has a first embedding dimension that is different from a second embedding dimension of the second embedding vector, and wherein the first embedding dimension is based at least in part on the first task. 
     
     
       13. The computer system of  claim 5 , wherein the task is a first task from multiple tasks of a multi-task layer (MTL) of an AI model that comprises the ML transformer, and wherein the ML transformer and the MTL are trained for the multiple tasks. 
     
     
       14. One or more non-transitory computer-readable storage media storing instructions that, upon execution on a computer system, cause the computer system to perform operations comprising:
 generating, by using a machine learning (ML) transformer, an embedding vector based at least in part on text, the embedding vector generated prior to performing natural language understanding (NLU) processing on the text; 
 storing the embedding vector and an association between the embedding vector and the text in a data store; 
 determining that a task is to be performed based at least in part on the NLU processing of the text; 
 sending, based at least in part on the task, a query to the data store; 
 receiving, in response to the query, the embedding vector from the data store based at least in part on the association between the embedding vector and the text; and 
 performing the task based at least in part on the embedding vector after being received from the data store. 
 
     
     
       15. The one or more non-transitory computer-readable storage media of  claim 14 , storing further instructions, that upon execution on the computer system, cause the computer system to perform additional operations comprising:
 determining a length of the text; and 
 editing, prior to inputting the text to the ML transformer, the text based at least in part on a comparison of the length to a length range, wherein the editing comprising a word addition or a word deletion. 
 
     
     
       16. The one or more non-transitory computer-readable storage media of  claim 15 , wherein the length range is set as a hyperparameter of the ML transformer during a training of the ML transformer. 
     
     
       17. The one or more non-transitory computer-readable storage media of  claim 14 , storing further instructions, that upon execution on the computer system, cause the computer system to perform additional operations comprising:
 generating a second embedding vector for a subset of words in the text, wherein the embedding vector is a first embedding vector generated for all words in the text. 
 
     
     
       18. The one or more non-transitory computer-readable storage media of  claim 17 , wherein the first embedding vector corresponds to a first task and has a first embedding dimension, wherein the second embedding vector corresponds to a second task and has a second embedding dimension different from the first embedding dimension. 
     
     
       19. The one or more non-transitory computer-readable storage media of  claim 14 , storing further instructions, that upon execution on the computer system, cause the computer system to perform additional operations comprising:
 storing, in the data store, metadata associated with the embedding vector, the metadata comprising: words that are included in the text and from which the embedding vector is generated, position information about the words in the text, and an identifier of an utterance from which the text is determined. 
 
     
     
       20. The one or more non-transitory computer-readable storage media of  claim 14 , storing further instructions, that upon execution on the computer system, cause the computer system to perform additional operations comprising:
 receiving, from a user device, audio data that corresponds to a user utterance detected by the user device; 
 generate the text by at least performing automatic speech recognition (ASR) on the audio data; 
 inputting the text to the ML transformer prior to performing the NLU processing on the text or completing the NLU processing of the text; and 
 inputting, after completion of the NLU processing, the embedding vector to a multi-task layer (MTL) of an AI model, the MTL trained to perform multiple tasks, wherein the AI model comprises the ML transformer.