Sequence-to-sequence speech recognition with latency threshold

Abstract

A computing system including one or more processors configured to receive an audio input. The one or more processors may generate a text transcription of the audio input at a sequence-to-sequence speech recognition model, which may assign a respective plurality of external-model text tokens to a plurality of frames included in the audio input. Each external-model text token may have an external-model alignment within the audio input. Based on the audio input, the one or more processors may generate a plurality of hidden states. Based on the plurality of hidden states, the one or more processors may generate a plurality of output text tokens. Each output text token may have a corresponding output alignment within the audio input. For each output text token, a latency between the output alignment and the external-model alignment may be below a predetermined latency threshold. The one or more processors may output the text transcription.

1 . A computing system comprising:
 one or more processors configured to:
 receive an audio input; 
 generate a text transcription of the audio input at a sequence-to-sequence speech recognition model that includes a trained external alignment model, the sequence-to-sequence speech recognition model being configured to at least:
 assign, via the trained external alignment model, a respective plurality of external-model text tokens to a plurality of frames included in the audio input, wherein each external-model text token has an external-model alignment within the audio input; 
 based at least in part on the audio input and the respective external-model alignments of the external-model text tokens, generate a plurality of output text tokens corresponding to the plurality of frames; 
 compute a latency between the plurality of external-model text tokens and the plurality of output text tokens based at least in part on differences between respective output boundaries of the output text tokens and corresponding external-model boundaries of the external-model text tokens, wherein:
 each output text token has a corresponding output alignment within the audio input; and 
 for each output text token, a latency between the output alignment and the external-model alignment is constrained to be below a predetermined latency threshold; and 
 
 
 output the text transcription including the plurality of output text tokens. 
   
     
     
         2 . The computing system of  claim 1 , wherein:
 the audio input is a streaming audio input received by the one or more processors over an input time interval; and   the one or more processors are configured to output the text transcription during the input time interval concurrently with receiving the audio input.   
     
     
         3 . The computing system of  claim 1 , wherein the one or more processors are further configured to pre-process the audio input at least in part by dividing the audio input into the plurality of frames. 
     
     
         4 . The computing system of  claim 1 , wherein, at the external alignment model, the one or more processors are further configured to assign the plurality of external-model text tokens to the frames as indicators of respective senone-level features included in the audio input. 
     
     
         5 . The computing system of  claim 1 , wherein the sequence-to-sequence speech recognition model includes one or more recurrent neural networks. 
     
     
         6 . The computing system of  claim 5 , wherein the external alignment model is a recurrent neural network. 
     
     
         7 . The computing system of  claim 5 , wherein the one or more recurrent neural networks include a trained encoder neural network and a trained decoder neural network. 
     
     
         8 . The computing system of  claim 7 , wherein the trained decoder neural network is a monotonic chunkwise attention model at which the one or more processors are further configured to:
 compute a plurality of monotonic energy activations based at least in part on a plurality of encoder outputs of the trained encoder neural network; and   compute a respective plurality of selection probabilities of the output text tokens based at least in part on the monotonic energy activations.   
     
     
         9 . The computing system of  claim 8 , wherein the one or more processors are configured to compute the output alignments of the output text tokens based at least in part on the plurality of selection probabilities. 
     
     
         10 . The computing system of  claim 1 , wherein the sequence-to-sequence speech recognition model further includes a one-dimensional convolutional layer. 
     
     
         11 . A method for use with a computing system, the method comprising:
 receiving an audio input;   generating a text transcription of the audio input at a sequence-to-sequence speech recognition model that includes a trained external alignment model, wherein generating the text transcription at the sequence-to-sequence speech recognition model includes:
 assigning, via the trained external alignment model, a respective plurality of external-model text tokens to a plurality of frames included in the audio input, wherein each external-model text token has an external-model alignment within the audio input; 
 based at least in part on the audio input and the respective external-model alignments of the external-model text tokens, generating a plurality of output text tokens corresponding to the plurality of frames; 
 computing a latency between the plurality of external-model text tokens and the plurality of output text tokens based at least in part on differences between respective output boundaries of the output text tokens and corresponding external-model boundaries of the external-model text tokens, wherein:
 each output text token has a corresponding output alignment within the audio input; and 
 for each output text token, a latency between the output alignment and the external-model alignment is constrained to be below a predetermined latency threshold; and 
 
   outputting the text transcription including the plurality of output text tokens.   
     
     
         12 . The method of  claim 11 , wherein:
 the audio input is a streaming audio input received over an input time interval; and   the text transcription is output during the input time interval concurrently with receiving the audio input.   
     
     
         13 . The method of  claim 11 , further comprising pre-processing the audio input at least in part by dividing the audio input into the plurality of frames. 
     
     
         14 . The method of  claim 11 , further comprising, at the external alignment model, assigning the plurality of external-model text tokens to the frames as indicators of respective senone-level features included in the audio input. 
     
     
         15 . The method of  claim 11 , wherein the sequence-to-sequence speech recognition model includes one or more recurrent neural networks. 
     
     
         16 . The method of  claim 15 , wherein the external alignment model is a recurrent neural network. 
     
     
         17 . The method of  claim 15 , wherein the one or more recurrent neural networks include a trained encoder neural network and a trained decoder neural network. 
     
     
         18 . The method of  claim 17 , wherein the trained decoder neural network is a monotonic chunkwise attention model, the method further comprising:
 computing a plurality of monotonic energy activations based at least in part on a plurality of encoder outputs of the trained encoder neural network; and   computing a respective plurality of selection probabilities of the output text tokens based at least in part on the monotonic energy activations.   
     
     
         19 . The method of  claim 11 , wherein the sequence-to-sequence speech recognition model further includes a one-dimensional convolutional layer. 
     
     
         20 . A computing system comprising:
 one or more processors configured to:
 receive an audio input; 
 pre-process the audio input at least in part by dividing the audio input into the plurality of frames; 
 generate a text transcription of the audio input at a sequence-to-sequence speech recognition model that includes a plurality of trained neural networks, the sequence-to-sequence speech recognition model being configured to at least:
 at a first trained neural network of the plurality of neural networks, assign a respective plurality of external-model text tokens to the plurality of frames, wherein each external-model text token has an external-model alignment within the audio input; and 
 at one or more additional trained neural networks of the plurality of trained neural networks:
 based at least in part on the audio input and the respective external-model alignments of the external-model text tokens, generate a plurality of output text tokens corresponding to the plurality of frames; 
 compute a latency between the plurality of external-model text tokens and the plurality of output text tokens based at least in part on differences between respective output boundaries of the output text tokens and corresponding external-model boundaries of the external-model text tokens, wherein: 
  each output text token has a corresponding output alignment within the audio input; and 
  for each output text token, a latency between the output alignment and the external-model alignment is constrained to be below a predetermined latency threshold; and 
 
 
 output the text transcription including the plurality of output text tokens.