Generating target sequences from input sequences using partial conditioning

Abstract

A system can be configured to perform tasks such as converting recorded speech to a sequence of phonemes that represent the speech, converting an input sequence of graphemes into a target sequence of phonemes, translating an input sequence of words in one language into a corresponding sequence of words in another language, or predicting a target sequence of words that follow an input sequence of words in a language (e.g., a language model). In a speech recognizer, the RNN system may be used to convert speech to a target sequence of phonemes in real-time so that a transcription of the speech can be generated and presented to a user, even before the user has completed uttering the entire speech input.

What is claimed is: 
     
       1. A method for generating a target sequence comprising a respective output at each of a plurality of output time steps from an input sequence comprising a respective input at each of a plurality of input time steps, the method comprising:
 for each block of a fixed number of input time steps in the input sequence:
 processing each input in the block of input time steps using an encoder neural network to generate a respective feature representation of the input; and 
 processing the feature representations for the inputs in the block and a preceding output at a preceding output time step using a transducer neural network to select a respective output for each of one or more output time steps immediately following the preceding output time step. 
 
 
     
     
       2. The method of  claim 1 , wherein, for the initial block in the input sequence, the preceding output at the preceding output time step is a placeholder start-of-sequence output. 
     
     
       3. The method of  claim 1 , wherein processing the feature representations for the inputs in the block and a preceding output at a preceding output time step using a transducer neural network to select a respective output for each of one or more output time steps immediately following the preceding output time step comprises selecting outputs until the selected output is a designated end-of-block output. 
     
     
       4. The method of  claim 3 , wherein processing the feature representations for the inputs in the block and the preceding output at a preceding output time step using the transducer neural network comprises:
 processing the feature representations for the inputs in the block and the preceding output using the transducer neural network to select a current output for a current output time step immediately following the preceding output time step; 
 when the current output is the designated end-of-block output, refraining from generating any more outputs for the block; and 
 when the current output is not the designated end-of-block output:
 processing the feature representations for the inputs in the block and the current output using the transducer neural network to select a next output for a next output time step immediately following the current output time step. 
 
 
     
     
       5. The method of  claim 1 , wherein processing the feature representations for the inputs in the block and a preceding output at a preceding output time step using a transducer neural network to select a respective output for each of one or more output time steps immediately following the preceding output time step comprises selecting outputs until a designation portion of an intermediate output of the transducer neural network indicates that the selected output is the last in the block. 
     
     
       6. The method of  claim 1 , wherein the transducer neural network is configured to, for a given block of input time steps and to select an output for a given output time step:
 process the output at an output time step immediately preceding the given output time step and a preceding context vector for the output time step immediately preceding the given output time step using a first subnetwork to update a current hidden state of the first subnetwork; 
 process the updated hidden state of the first subnetwork and the feature representations for the inputs in the given block of input time steps using a context subnetwork to determine a current context vector; 
 process the current context vector and the updated hidden state of the first subnetwork using a second subnetwork to update a current hidden state of the second subnetwork; and 
 process the current hidden state of the second subnetwork using a softmax layer to generate a respective score for each output in a dictionary of possible outputs. 
 
     
     
       7. The method of  claim 6 , wherein the context subnetwork is a recurrent neural network. 
     
     
       8. The method of  claim 1 , wherein the input sequence is a speech sequence and the target sequence is a sequence of phonemes representing the speech sequence. 
     
     
       9. A computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations for generating a target sequence comprising a respective output at each of a plurality of output time steps from an input sequence comprising a respective input at each of a plurality of input time steps, the operations comprising:
 for each block of a fixed number of input time steps in the input sequence:
 processing each input in the block of input time steps using an encoder neural network to generate a respective feature representation of the input; and 
 processing the feature representations for the inputs in the block and a preceding output at a preceding output time step using a transducer neural network to select a respective output for each of one or more output time steps immediately following the preceding output time step. 
 
 
     
     
       10. The computer storage medium of  claim 9 , wherein, for the initial block in the input sequence, the preceding output at the preceding output time step is a placeholder start-of-sequence output. 
     
     
       11. The computer storage medium of  claim 9 , wherein processing the feature representations for the inputs in the block and a preceding output at a preceding output time step using a transducer neural network to select a respective output for each of one or more output time steps immediately following the preceding output time step comprises selecting outputs until the selected output is a designated end-of-block output. 
     
     
       12. The computer storage medium of  claim 9 , wherein the transducer neural network is configured to, for a given block of input time steps and to select an output for a given output time step:
 process the output at an output time step immediately preceding the given output time step and a preceding context vector for the output time step immediately preceding the given output time step using a first subnetwork to update a current hidden state of the first subnetwork; 
 process the updated hidden state of the first subnetwork and the feature representations for the inputs in the given block of input time steps using a context subnetwork to determine a current context vector; 
 process the current context vector and the updated hidden state of the first subnetwork using a second subnetwork to update a current hidden state of the second subnetwork; and 
 process the current hidden state of the second subnetwork using a softmax layer to generate a respective score for each output in a dictionary of possible outputs. 
 
     
     
       13. A method for generating a target sequence comprising a respective output at each of a plurality of output time steps from an input sequence comprising a respective input at each of a plurality of input time steps, the method comprising:
 for each block of a fixed number of input time steps in the input sequence:
 processing each input in the block of input time steps using a first set of neural network operations to generate a respective feature representation of the input; and 
 processing the feature representations for the inputs in the block and a preceding output at a preceding output time step using a second set of neural network operations to select a respective output for each of one or more output time steps immediately following the preceding output time step. 
 
 
     
     
       14. The method of  claim 13 , wherein, for the initial block in the input sequence, the preceding output at the preceding output time step is a placeholder start-of-sequence output. 
     
     
       15. The method of  claim 13 , wherein processing the feature representations for the inputs in the block and a preceding output at a preceding output time step using the second set of neural network operations to select a respective output for each of one or more output time steps immediately following the preceding output time step comprises selecting outputs until the selected output is a designated end-of-block output. 
     
     
       16. The method of  claim 15 , wherein processing the feature representations for the inputs in the block and the preceding output at a preceding output time step using the second set of neural network operations comprises:
 processing the feature representations for the inputs in the block and the preceding output using the second set of neural network operations to select a current output for a current output time step immediately following the preceding output time step; 
 when the current output is the designated end-of-block output, refraining from generating any more outputs for the block; and 
 when the current output is not the designated end-of-block output:
 processing the feature representations for the inputs in the block and the current output using the second set of neural network operations to select a next output for a next output time step immediately following the current output time step. 
 
 
     
     
       17. The method of  claim 13 , wherein processing the feature representations for the inputs in the block and a preceding output at a preceding output time step using the second set of neural network operations to select a respective output for each of one or more output time steps immediately following the preceding output time step comprises selecting outputs until a designation portion of an intermediate output of the second set of neural network operations indicates that the selected output is the last in the block. 
     
     
       18. The method of  claim 13 , wherein the input sequence is a speech sequence and the target sequence is a sequence of phonemes representing the speech sequence. 
     
     
       19. The method of  claim 13 , wherein the first set of neural network operations comprise an encoder recurrent neural network and the second set of neural network operations comprise a transducer recurrent neural network.