Joint noise and echo suppression for two-way audio communication enhancement

Abstract

Joint noise and echo suppression may be performed for enhancing two-way audio communications. Audio data is captured at a communication device and audio data transmitted to the communication device from another communication device are used as input features to a trained machine learning model that uses the transmitted audio data as a reference signal to eliminate residual echo in the captured audio data when also suppressing noise in the captured audio data.

What is claimed is: 
     
       1. A system, comprising:
 at least one processor; and 
 a memory, storing program instructions that when executed by the at least one processor, cause the at least one processor to implement an audio enhancement system, configured to:
 receive, via an interface for the audio enhancement system, first audio data captured by a microphone at a first communication device as part of a two-way audio communication between the first communication device and a second communication device; 
 receive second audio data transmitted from the second communication device to the first communication device for playback through a speaker at the first communication device as part of the two-way audio communication; 
 apply a machine learning model trained to determine respective gain values for a plurality of different spectrum bands of the first audio data to suppress noise and suppress echo captured in the first audio data from playback of the second audio data through a speaker at the first communication device, wherein the machine learning model accepts respective input features extracted from the second audio data as a reference signal and extracted from the first audio data based on respective representations of the second audio data and the first audio data in respective sets of frequency bands; and 
 apply an envelope post-filter that individually modifies the respective gain values according to a monotonically increasing function applied to the respective gain values; 
 perform an inverse transform on the plurality of different spectrum bands with the respectively modified gain values to generate an enhanced version of the first audio data; and 
 send the enhanced version of the first audio data to the second communication device for playback at the second communication device. 
 
 
     
     
       2. The system of  claim 1 , wherein the audio enhancement system is further configured to apply an acoustic echo canceller to the first audio data prior to providing the first audio data as the input features to the machine learning model. 
     
     
       3. The system of  claim 1 , wherein the system is implemented as part of the first communication device, and wherein the enhanced version of the first audio data is sent to the second communication device via a communication service implemented as part of a provider network that transmits the enhanced version of the first audio data to the second communication device over a network connection. 
     
     
       4. The system of  claim 1 , wherein the audio enhancement system is implemented as part of a communication service offered as part of a provider network, and wherein the enhanced version of the first audio data is sent to the second communication device over a network connection between the communication service and the second communication device. 
     
     
       5. A method, comprising:
 receiving, via an interface for an audio enhancement system, first audio data captured by a microphone at a first communication device as part of a two-way communication between the first communication device and a second communication device; 
 receiving, via the interface for the audio enhancement system, second audio data transmitted from the second communication device to the first communication device as part of the two-way communication; 
 applying, by the audio enhancement system, a machine learning model trained to determine respective gain values for a plurality of different spectrum bands of the first audio data to suppress noise and suppress echo captured in the first audio data from playback of the second audio data through a speaker at the first communication device, wherein the machine learning model accepts respective input features extracted from the second audio data as a reference signal and extracted from the first audio data based on respective representations of the second audio data and the first audio data in respective sets of frequency bands; and 
 applying, by the audio enhancement system, an envelope post-filter that individually modifies the respective gain values according to a monotonically increasing function applied to the respective gain values; and 
 providing, by the audio enhancement system, an enhanced version of the first audio data generated, based, at least in part, on the respectively modified gain values for the plurality of different spectrum bands. 
 
     
     
       6. The method of  claim 5 , further comprising applying an acoustic echo canceller to the first audio data prior to providing the first audio data as the input features to the machine learning model. 
     
     
       7. The method of  claim 5 , wherein the second audio data comprises two or more channels for playback through the speaker and wherein the two or more channels of the second audio data are combined as the reference signal. 
     
     
       8. The method of  claim 5 , wherein the two or more channels are combined through a downmixing technique that averages the two or more channels. 
     
     
       9. The method of  claim 5 , further comprising selecting, by the audio enhancement system, a version of the machine learning model out of a plurality of versions of the machine learning model to apply based, at least in part, on a computational capacity of the audio enhancement system, as part of initializing the audio enhancement system. 
     
     
       10. The method of  claim 5 , wherein the audio enhancement system is implemented as part of a communication service offered as part of a provider network, and wherein the enhanced version of the first audio data is sent to the second communication device over a network connection between the communication service and the second communication device. 
     
     
       11. The method of  claim 5 , wherein the audio enhancement system is implemented as part of the first communication device, and wherein the enhanced version of the first audio data is sent to the second communication device via a communication service implemented as part of a provider network that transmits the enhanced version of the first audio data to the second communication device over a network connection. 
     
     
       12. The method of  claim 5 , wherein the two-way communication is a video communication, wherein the first audio data and the second audio data is captured along with corresponding two-way video data. 
     
     
       13. One or more non-transitory, computer-readable storage media, storing program instructions that when executed on or across one or more computing devices cause the one or more computing devices to implement:
 receiving, via an interface for an audio enhancement system, first audio data captured by a microphone at a first communication device as part of a two-way communication between the first communication device and a second communication device; 
 receiving, via the interface for the audio enhancement system, second audio data transmitted from the second communication device to the first communication device as part of the two-way communication; 
 applying, by the audio enhancement system, a machine learning model trained to determine respective gain values for a plurality of different spectrum bands of the first audio data to suppress noise and suppress echo captured in the first audio data from playback of the second audio data through a speaker at the first communication device, wherein the machine learning model accepts as respective input features the second audio data as a reference signal and the first audio data based on respective representations of the second audio data and the first audio data in respective sets of frequency bands; and 
 applying, by the audio enhancement system, an envelope post-filter that individually modifies the respective gain values according to a monotonically increasing function applied to the respective gain values; and 
 performing, by the audio enhancement system, an inverse transform on the plurality of different spectrum bands with the respectively modified gain values to generate an enhanced version of the first audio data. 
 
     
     
       14. The one or more non-transitory, computer-readable storage media of  claim 13 , storing further program instructions that when executed by the one or more computing devices, cause the one or more computing devices to further implement applying, by the audio enhancement system, an acoustic echo canceller to the first audio data prior to providing the first audio data as the input features to the machine learning model. 
     
     
       15. The one or more non-transitory, computer-readable storage media of  claim 13 , wherein the second audio data comprises two or more channels for playback through the speaker and wherein the two or more channels of the second audio data are combined as the reference signal. 
     
     
       16. The one or more non-transitory, computer-readable storage media of  claim 15 , wherein the two or more channels are combined by concatenating the two or more channels. 
     
     
       17. The one or more non-transitory, computer-readable storage media of  claim 13 , wherein the respective sets of frequency bands are psychoacoustically-based bands. 
     
     
       18. The one or more non-transitory, computer-readable storage media of  claim 17 , wherein the respective sets of frequency bands that are psychoacoustically-based bands are in a same equivalent rectangular bandwidth scale. 
     
     
       19. The one or more non-transitory, computer-readable storage media of  claim 13 , wherein the audio enhancement system is implemented as part of a communication service offered as part of a provider network, and wherein the enhanced version of the first audio data is sent to the second communication device over a network connection between the communication service and the second communication device. 
     
     
       20. The one or more non-transitory, computer-readable storage media of  claim 13 , wherein the two-way communication is a video communication, wherein the first audio data and the second audio data is captured along with corresponding two-way video data.