Mixing decision controlling decode decision

Abstract

Methods, systems, and apparatus are provided for combining (e.g., mixing) audio signals received from a plurality of participants communicating with each other during a communication session (e.g., an audio conference) based on voice-activity-detection (VAD) data contained in extended headers of Real-time Transport Protocol (RTP) packets. An audio mixing apparatus receives RTP packets from connected clients and extracts the VAD data included in the extended RTP headers to render a mixing decision. Once a mixing decision has been made, the audio frames to be mixed are decoded while other received frames are discarded, thereby preventing processing resources from being wasted to decode frames that are never used.

We claim: 
     
         1 . A method for mixing audio signals comprising:
 receiving, at an audio mixing apparatus, audio packets from a plurality of clients in communication with the audio mixing apparatus;   retrieving audio level data contained in an extended packet header of each of the received audio packets;   selecting audio frames to be mixed based on the audio level data retrieved from the extended packet headers of the audio packets;   decoding the audio frames selected to be mixed; and   generating a mixed audio stream by mixing the decoded audio frames.   
     
     
         2 . The method of  claim 1 , further comprising:
 responsive to receiving the audio packets from the plurality of clients, determining, for each audio frame contained in the audio packets, whether audio level data can be extracted from the audio frame without decoding the audio frame.   
     
     
         3 . The method of  claim 2 , further comprising:
 responsive to determining that audio level data can be extracted from the audio frame without decoding the audio frame, extracting audio level data from the audio frame;   storing the audio level data extracted from the audio frame in an audio level data set; and   storing the audio frame in an encoded audio frames set.   
     
     
         4 . The method of  claim 3 , further comprising:
 responsive to determining that no audio level data can be extracted from the audio frame without decoding the audio frame, decoding the audio frame;   performing audio level processing on the decoded audio frame to obtain audio level data for the decoded audio frame;   storing the audio level data obtained for the decoded audio frame in the audio level data set; and   storing the decoded audio frame in a decoded audio frames set.   
     
     
         5 . The method of  claim 4 , further comprising:
 selecting a group of encoded audio frames from the encoded audio frames set and a group of decoded audio frames from the decoded audio frames set;   retrieving, from the audio level data set, the audio level data stored for the group of encoded audio frames and the group of decoded audio frames;   applying a mixing decision algorithm based on the audio level data retrieved from the audio level data set; and   determining a set of encoded and decoded audio frames to be mixed based on the applied mixing decision algorithm.   
     
     
         6 . The method of  claim 1 , further comprising storing the received audio packets in a buffer of the audio mixing apparatus. 
     
     
         7 . The method of  claim 1 , further comprising discarding the audio frames not selected to be mixed. 
     
     
         8 . The method of  claim 1 , wherein the audio packets received from the plurality of clients are Real-Time Transport Protocol (RTP) packets. 
     
     
         9 . The method of  claim 1 , wherein the audio level data contained in the extended packet header of each of the received audio packets includes voice data corresponding to one of the plurality of clients. 
     
     
         10 . An audio mixing apparatus configured to perform operations comprising:
 receiving audio packets from a plurality of clients in communication with the audio mixing apparatus;   retrieving audio level data contained in an extended packet header of each of the received audio packets;   selecting audio frames to be mixed based on the audio level data retrieved from the extended packet headers of the audio packets;   decoding the audio frames selected to be mixed; and   generating a mixed audio stream by mixing the decoded audio frames.   
     
     
         11 . The audio mixing apparatus of  claim 10 , further configured to perform operations comprising, responsive to receiving the audio packets from the plurality of clients, determining, for each audio frame contained in the audio packets, whether audio level data can be extracted from the audio frame without decoding the audio frame. 
     
     
         12 . The audio mixing apparatus of  claim 11 , further configured to perform operations comprising:
 responsive to determining that audio level data can be extracted from the audio frame without decoding the audio frame, extracting audio level data from the audio frame;   storing the audio level data extracted from the audio frame in an audio level data set; and   storing the audio frame in an encoded audio frames set.   
     
     
         13 . The audio mixing apparatus of  claim 12 , further configured to perform operations comprising:
 responsive to determining that no audio level data can be extracted from the audio frame without decoding the audio frame, decoding the audio frame;   performing audio level processing on the decoded audio frame to obtain audio level data for the decoded audio frame;   storing the audio level data obtained for the decoded audio frame in the audio level data set; and   storing the decoded audio frame in a decoded audio frames set.   
     
     
         14 . The audio mixing apparatus of  claim 13 , further configured to perform operations comprising:
 selecting a group of encoded audio frames from the encoded audio frames set and a group of decoded audio frames from the decoded audio frames set;   retrieving, from the audio level data set, the audio level data stored for the group of encoded audio frames and the group of decoded audio frames;   applying a mixing decision algorithm based on the audio level data retrieved from the audio level data set; and   determining a set of encoded and decoded audio frames to be mixed based on the applied mixing decision algorithm.   
     
     
         15 . The audio mixing apparatus of  claim 10 , further configured to perform operations comprising storing the received audio packets in a buffer of the audio mixing apparatus. 
     
     
         16 . The audio mixing apparatus of  claim 10 , further configured to perform operations comprising discarding the audio frames not selected to be mixed. 
     
     
         17 . The audio mixing apparatus of  claim 10 , wherein the audio packets received from the plurality of clients are Real-Time Transport Protocol (RTP) packets. 
     
     
         18 . The audio mixing apparatus of  claim 10 , wherein the audio level data contained in the extended packet header of each of the received audio packets includes voice data corresponding to one of the plurality of clients.