Audio-visual speech recognition with scattering operators

Abstract

Aspects described herein are directed towards methods, computing devices, systems, and computer-readable media that apply scattering operations to extracted visual features of audiovisual input to generate predictions regarding the speech status of a subject. Visual scattering coefficients generated according to one or more aspects described herein may be used as input to a neural network operative to generate the predictions regarding the speech status of the subject. Predictions generated based on the visual features may be combined with predictions based on audio input associated with the visual features. In some embodiments, the extracted visual features may be combined with the audio input to generate a combined feature vector for use in generating predictions.

What is claimed is: 
     
       1. A method comprising:
 receiving, by a computing device, audiovisual input comprising audio input and video input associated with a subject; 
 extracting, by the computing device, visual features from the video input; 
 applying, by the computing device, a scattering operation to the extracted visual features to generate a vector of scattering coefficients; 
 providing the vector of scattering coefficients as input to a first neural network for visual processing; 
 providing the audio input to a second neural network for audio processing; 
 combining, by the computing device, a first output of the first neural network with a second output of the second neural network to generate a fused audiovisual feature vector based on the audiovisual input; 
 providing the fused audiovisual feature vector to a third neural network for audiovisual processing; and 
 generating, by the computing device and using the third neural network, a first prediction regarding a speech status of the subject based on the fused audiovisual feature vector. 
 
     
     
       2. The method of  claim 1 , wherein providing the vector of scattering coefficients as input to the first neural network for visual processing comprises:
 normalizing the vector of scattering coefficients to generate a first normalized vector of scattering coefficients; 
 aggregating a plurality of normalized vectors of scattering coefficients including the first normalized vector of scattering coefficients to generate a set of aggregated visual feature vectors; and 
 providing the set of aggregated visual feature vectors to the first neural network for visual input processing. 
 
     
     
       3. The method of  claim 1 , wherein the video input is sampled at a first frequency and the audio input is sampled at a different second frequency. 
     
     
       4. The method of  claim 1 , wherein the vector of scattering coefficients is in a first dimensional space, and wherein applying the scattering operation to the extracted visual features to generate the vector of scattering coefficients comprises:
 applying the scattering operation to the extracted visual features to generate a second vector of scattering coefficients in a second dimensional space; and 
 projecting the second vector of scattering coefficients into the first dimensional space to generate the vector of scattering coefficients in the first dimensional space, 
 wherein the second dimensional space is of a higher dimensionality than the first dimensional space. 
 
     
     
       5. The method of  claim 1 , wherein applying the scattering operation to the extracted visual features to generate a vector of scattering coefficients comprises generating first order scattering coefficients and second order scattering coefficients. 
     
     
       6. The method of  claim 1 , further comprising:
 generating, by the computing device and using the first neural network, a second prediction regarding the speech status of the subject based on the vector of scattering coefficients; and 
 generating, by the computing device and using the second neural network, a third prediction regarding the speech status of the subject based on the audio input; and 
 combining the first prediction, the second prediction, and the third prediction to generate a combined prediction regarding the speech status of the subject. 
 
     
     
       7. The method of  claim 1 , wherein the first prediction regarding the speech status of the subject is used to recognize speech content of the audiovisual input. 
     
     
       8. The method of  claim 1 , further comprising:
 determining, based on the first prediction regarding the speech status of the subject, a synchrony state of the audio input and the video input, wherein the synchrony state indicates whether the audio input is in-sync with the video input. 
 
     
     
       9. The method of  claim 1 , further comprising:
 determining, based on the first prediction regarding the speech status of the subject, whether speech in the audio input originates from a foreground source or a background source. 
 
     
     
       10. One or more non-transitory computer readable media comprising instructions that, when executed by one or more processors, cause the one or more processors to perform steps comprising:
 receiving audiovisual input comprising audio input and video input associated with a subject; 
 extracting visual features from the video input; 
 applying a scattering operation to the extracted visual features to generate a vector of scattering coefficients in a first dimensional space; 
 providing the vector of scattering coefficients as input to a first neural network for visual processing; 
 providing the audio input to a second neural network for audio processing; 
 combining a first output of the first neural network with a second output of the second neural network to generate a fused audiovisual feature vector based on the audiovisual input; 
 providing the fused audiovisual feature vector to a third neural network for audiovisual processing; and 
 generating, using the third neural network, a prediction regarding a speech status of the subject based on the fused audiovisual feature vector. 
 
     
     
       11. The one or more non-transitory computer readable media of  claim 10 , wherein providing the vector of scattering coefficients as input to the first neural network for visual processing comprises:
 normalizing the vector of scattering coefficients to generate a first normalized vector of scattering coefficients; 
 aggregating a plurality of normalized vectors of scattering coefficients including the first normalized vector of scattering coefficients to generate a set of aggregated visual feature vectors; and 
 providing the set of aggregated visual feature vectors to the first neural network for visual input processing. 
 
     
     
       12. The one or more non-transitory computer readable media of  claim 10 , wherein the video input is sampled at a first frequency and the audio input is sampled at a different second frequency. 
     
     
       13. The one or more non-transitory computer readable media of  claim 10 , wherein applying the scattering operation to the extracted visual features to generate the vector of scattering coefficients comprises:
 applying the scattering operation to the extracted visual features to generate a second vector of scattering coefficients in a second dimensional space; and 
 projecting the second vector of scattering coefficients into the first dimensional space to generate the vector of scattering coefficients in the first dimensional space, 
 wherein the second dimensional space is of a higher dimensionality than the first dimensional space. 
 
     
     
       14. The one or more non-transitory computer readable media of  claim 10 , wherein the prediction regarding the speech status of the subject is used to recognize speech content of the audiovisual input. 
     
     
       15. The one or more non-transitory computer readable media of  claim 10 , wherein the prediction regarding the speech status of the subject is used to determine a synchrony state of the audio input and the video input, wherein the synchrony state indicates whether the audio input is in-sync with the video input. 
     
     
       16. The one or more non-transitory computer readable media of  claim 10 , wherein the prediction regarding the speech status of the subject is used to determine whether speech in the audio input originates from a foreground source or a background source. 
     
     
       17. An apparatus comprising:
 one or more processors; and 
 memory storing instructions that, when executed by the one or more processors, cause the apparatus to:
 receive audiovisual input comprising audio input and video input associated with a subject; 
 extract visual features from the video input; 
 apply a scattering operation to the extracted visual features to generate a vector of scattering coefficients; 
 provide the vector of scattering coefficients as input to a first neural network for visual processing; 
 provide the audio input to a second neural network for audio processing; 
 combine a first output of the first neural network with a second output of the second neural network to generate a fused audiovisual feature vector based on the audiovisual input; 
 provide the fused audiovisual feature vector to a third neural network for audiovisual processing; and 
 generate, and using the third neural network, a first prediction regarding a speech status of the subject based on the fused audiovisual feature vector. 
 
 
     
     
       18. The apparatus of  claim 17 , wherein the video input is sampled at a first frequency and the audio input is sampled at a different second frequency. 
     
     
       19. The apparatus of  claim 17 , wherein the vector of scattering coefficients is in a first dimensional space, and wherein the instructions, when executed by the one or more processors, further cause the apparatus to:
 generate, using the first neural network, a second prediction regarding the speech status of the subject based on the vector of scattering coefficients; and 
 generate, using the second neural network, a third prediction regarding the speech status of the subject based on the audio input; and 
 combine the first prediction, the second prediction, and the third prediction to generate a combined prediction regarding the speech status of the subject. 
 
     
     
       20. The apparatus of  claim 17 , wherein the instructions, when executed by the one or more processors, further cause the apparatus to:
 determine, based on the first prediction regarding the speech status of the subject, whether speech in the audio input originates from a foreground source or a background source.