Audio loudness control method and system based on signal analysis and deep learning

Abstract

The present disclosure relates to a method and system for controlling loudness of an audio based on signal analysis and deep learning. The method includes analyzing an audio characteristic in a frame level based on signal analysis, analyzing the audio characteristic in the frame level based on learning, and controlling loudness of the audio in the frame level, by combining the analysis results. Accordingly, reliability of audio characteristic analysis can be enhanced and audio loudness can be optimally controlled.

What is claimed is: 
     
       1. A method of controlling a loudness of an audio signal that comprises one or more frames, the method comprising:
 analyzing, at a computing device comprising one or more processors, a first frame of the audio signal using a deep-learning based analysis unit of the computing device, to determine a genre of the audio signal and to determine a reference loudness based on the genre; 
 analyzing, at the computing device, the first frame of the audio signal using a non- deep-learning based analysis technique to measure a loudness of the first frame; 
 determining a first category of the first frame based on the loudness measured using the non-deep-learning based analysis technique and the reference loudness determined using the deep-learning based analysis unit of the computing device, wherein the first category indicates that the first frame is a voice frame when the measured loudness is greater than the reference loudness, and wherein the first category indicates that the first frame is a silence frame when the measured loudness is less than the reference loudness; 
 analyzing, at the computing device, the first frame of the audio signal using the deep-learning based analysis to obtain unit of the computing device to obtain a second category of the first frame, wherein the second category indicates that the first frame is a foreground sound frame, a background sound frame, or a silence frame; 
 determining, at the computing device, an importance level of the first frame based on a combination of the first category and the second category, wherein the importance level of the first frame is higher in the combination of the first frame being the voice frame in the first category and the foreground sound frame in the second category than in the combination of the first frame being the voice frame in the first category and the background sound frame in the second category, and wherein the importance level of the first frame is higher in the combination of the first frame being the voice frame in the first category and the background sound frame in the second category than in the combination of the first frame being the silence frame in the first category and the background sound frame in the second category; 
 determining, at the computing device, an adjusted loudness of the first frame according to the determined importance level of the first frame such that the higher the importance level, the louder the first frame; and 
 causing the computing device to generate an audio output based on the adjusted loudness of the first frame. 
 
     
     
       2. The method of  claim 1 , wherein the second category indicates that the first frame is a foreground sound frame when a voice signal outweighs a noise signal in the first frame. 
     
     
       3. The method of  claim 1 , wherein the second category indicates that the first frame is a background sound frame when a noise signal outweighs a voice signal in the first frame. 
     
     
       4. The method of  claim 1 , further comprising:
 measuring a loudness of the audio output; and 
 additionally adjusting a loudness of the one or more frames of the audio signal when an average of the measured loudness exceeds a predetermined target loudness. 
 
     
     
       5. A system for controlling a loudness of an audio signal that comprises one or more frames, the system comprising:
 at least one processor; and 
 at least one memory storing instructions, the instructions configured to cause the at least one processor to perform: 
 analyzing a first frame of the audio signal using a deep-learning based analysis unit of the at least one processor to determine a genre of the audio signal and to determine a reference loudness based on the genre; 
 analyzing the first frame of the audio signal using a non-deep-learning based analysis technique to measure a loudness of the first frame; 
 determining a first category of the first frame based on the loudness measured using the non-deep-learning based analysis technique and the reference loudness determined using the deep-learning based analysis unit of the at least one processor, wherein the first category indicates that the first frame is a voice frame when the measured loudness is greater than the reference loudness, and wherein the first category indicates that the first frame is a silence frame when the measured loudness is less than the reference loudness; 
 analyzing the first frame of the audio signal using the deep-learning based analysis unit of the at least one processor to obtain a second category of the first frame, wherein the second category indicates that the first frame is a foreground sound frame, a background sound frame, or a silence frame; 
 determining an importance level of the first frame based on a combination of the first category and the second category, wherein the importance level of the first frame is higher in the combination of the first frame being the voice frame in the first category and the foreground sound frame in the second category than in the combination of the first frame being the voice frame in the first category and the background sound frame in the second category, and wherein the importance level of the first frame is higher in the combination of the first frame being the voice frame in the first category and the background sound frame in the second category than in the combination of the first frame being the silence frame in the first category and the background sound frame in the second category; 
 determining an adjusted loudness of the first frame according to the determined importance level of the first frame such that the higher the importance level, the louder the first frame; and 
 causing to generate an audio output based on the adjusted loudness of the first frame.