US12062369B2ActiveUtilityA1
Real-time dynamic noise reduction using convolutional networks
Est. expirySep 25, 2040(~14.2 yrs left)· nominal 20-yr term from priority
G10L 15/16G10L 25/30G10L 15/20G10L 21/0208
72
PatentIndex Score
1
Cited by
66
References
25
Claims
Abstract
A system, method and computer readable medium for dynamic noise reduction in a voice call. The system includes an encoder having a short-time Fourier transform module to determine a magnitude spectrum and a phase spectrum of an input audio signal, including speech and dynamic noise. A separator coupled to the encoder comprises a temporal convolution network (TCN) used to develop a separation mask using the magnitude spectrum as input. The TCN is trained using a frequency SNR function used to calculate loss during training. A mixer is coupled to the separator to multiply the separation mask with the magnitude spectrum to separate the speech from the dynamic noise to obtain a denoise magnitude spectrum. A decoder coupled to the mixer and the encoder includes an inverse short-time Fourier transform module to reconstruct the input audio signal without the dynamic noise using the denoise magnitude spectrum and the phase spectrum.
Claims
exact text as granted — not AI-modifiedWhat is claimed is:
1. A dynamic noise reduction system, comprising:
an encoder comprising a short-time Fourier transform module to determine a magnitude spectrum and a phase spectrum of an input audio signal, the input audio signal comprising speech and dynamic noise;
a separator, coupled to the encoder, comprising a temporal convolution network (TCN) used to develop a separation mask using the magnitude spectrum as input, wherein the TCN is trained using a frequency SNR cost function used to calculate loss during training, and wherein the TCN merges non-causal convolution layers with causal convolution layers to form a hybrid TCN architecture;
a mixer, coupled to the separator, to multiply the separation mask with the magnitude spectrum to separate the speech from the dynamic noise to obtain a denoise magnitude spectrum; and
a decoder, coupled to the mixer and the encoder, comprising an inverse short-time Fourier transform module to reconstruct the input audio signal without the dynamic noise using the denoise magnitude spectrum and the phase spectrum.
2. The system of claim 1 , wherein the dynamic noise reduction system operates in real-time, implementing a grouping mechanism to collect a pre-determined number of frames as a group of frames and perform an inference on the group of frames simultaneously to reduce computational requirements.
3. The system of claim 1 , wherein the TCN comprises at least one stack of 1-D dilated convolution blocks that repeat n times.
4. The system of claim 3 , wherein the at least one stack of 1-D dilated convolution blocks includes five (5) convolution layers that repeat two times.
5. The system of claim 1 , wherein the frequency SNR cost function includes target signal power that prevents an estimated error from being affected by varying signal levels during training.
6. The system of claim 1 , wherein the frequency SNR cost function includes a logarithmic scale to balance quiet and loud magnitudes.
7. The system of claim 1 , wherein the frequency SNR cost function comprises:
fSNR
(
X
,
X
^
)
=
10
*
log
(
∑
k
,
n
X
k
,
n
2
∑
k
,
n
(
X
^
-
X
)
2
+
ϵ
+
ϵ
)
,
with
X comprising target signal magnitude SFTF, {circumflex over (X)} comprising estimated signal magnitude STFT, k comprising STFT bins, n comprising STFT frames, and ε comprising numerical stability.
8. The system of claim 1 , wherein the input audio signal comprises an audio signal from a voice call.
9. The system of claim 1 , wherein the dynamic noise reduction system is executable on small form factor devices capable of voice calls.
10. A method for dynamic noise reduction, comprising:
receiving, by an encoder, an input audio signal, the input audio signal including speech and dynamic noise;
performing, by the encoder, a short-time Fourier transform on the audio signal to generate a magnitude spectrum and a phase spectrum;
estimating, by a temporal convolution network (TCN), a separation mask based on the magnitude spectrum using deep learning, wherein the TCN is trained using a frequency SNR cost function used to calculate loss during training, and wherein the TCN comprises non-causal convolution layers merged with causal convolution layers;
mixing the separation mask with the magnitude spectrum to generate a denoise magnitude spectrum; and
performing, by a decoder, an inverse short-time Fourier transform using the denoise magnitude spectrum and the phase spectrum to reconstruct the input audio signal without the dynamic noise.
11. The method of claim 10 , wherein the cost function comprises:
fSNR
(
X
,
X
^
)
=
10
*
log
(
∑
k
,
n
X
k
,
n
2
∑
k
,
n
(
X
^
-
X
)
2
+
ϵ
+
ϵ
)
,
with
X comprising target signal magnitude SFTF, {circumflex over (X)} comprising estimated signal magnitude STFT, k comprising STFT bins, n comprising STFT frames, and ε comprising numerical stability.
12. The method of claim 10 , wherein the frequency SNR cost function includes target signal power that prevents an estimated error from being affected by varying signal levels during training.
13. The method of claim 10 , wherein the frequency SNR cost function includes a logarithmic scale to balance quiet and loud magnitudes.
14. The method of claim 10 , wherein the dynamic noise reduction method operates in real-time, implementing a grouping mechanism to collect a pre-determined number of frames as a group of frames and perform an inference on the group of frames simultaneously to reduce computational requirements.
15. The method of claim 10 , wherein the dynamic noise reduction method is executable on small form factor devices capable of voice calls.
16. The method of claim 10 , wherein the input audio signal comprises an audio signal from a voice call.
17. The method of claim 10 , wherein the TCN comprises at least one stack of 1-D dilated convolution blocks that repeat n times to estimate the separation mask using the deep learning.
18. At least one non-transitory computer readable medium, comprising a set of instructions, which when executed by one or more computing devices, cause the one or more computing devices to:
receive, by an encoder, an input audio signal, the input audio signal including speech and dynamic noise;
perform, by the encoder, a short-time Fourier transform on the audio signal to generate a magnitude spectrum and a phase spectrum;
estimate, by a temporal convolution network (TCN), a separation mask based on the magnitude spectrum using deep learning, wherein the TCN is trained using a frequency SNR cost function used to calculate loss during training, and wherein the TCN comprises non-causal convolution layers merged with causal convolution layers;
mix the separation mask with the magnitude spectrum to generate a denoise magnitude spectrum; and
perform, by a decoder, an inverse short-time Fourier transform using the denoise magnitude spectrum and the phase spectrum to reconstruct the input audio signal without the dynamic noise.
19. The at least one non-transitory computer readable medium of claim 18 , wherein the cost function comprises:
fSNR
(
X
,
X
^
)
=
10
*
log
(
∑
k
,
n
X
k
,
n
2
∑
k
,
n
(
X
^
-
X
)
2
+
ϵ
+
ϵ
)
,
with X comprising target signal magnitude SFTF, {circumflex over (X)} comprising estimated signal magnitude STFT, k comprising STFT bins, n comprising STFT frames, and ε comprising numerical stability.
20. The at least one non-transitory computer readable medium of claim 18 , wherein the frequency SNR cost function includes target signal power that prevents an estimated error from being affected by varying signal levels during training.
21. The at least one non-transitory computer readable medium of claim 18 , wherein the frequency SNR cost function includes a logarithmic scale to balance quiet and loud magnitudes.
22. The at least one non-transitory computer readable medium of claim 18 , wherein the TCN comprises at least one stack of 1-D dilated convolution blocks that repeat n times to estimate the separation mask using the deep learning.
23. The at least one non-transitory computer-readable medium of claim 22 , wherein the at least one stack of 1-D dilated convolution blocks includes five (5) convolution layers that repeat two times.
24. The at least one non-transitory computer readable medium of claim 18 , wherein dynamic noise reduction operates in real-time, implementing a grouping mechanism to collect a pre- determined number of frames as a group of frames and perform an inference on the group of frames simultaneously to reduce computational requirements.
25. The at least one non-transitory computer readable medium of claim 18 , wherein the input audio signal comprises an audio signal from a voice call.Cited by (0)
No later patents cite this yet.
References (0)
No backward citations on record.