Augmented multi-tier classifier for multi-modal voice activity detection

Abstract

Disclosed herein are systems, methods, and computer-readable storage media for detecting voice activity in a media signal in an augmented, multi-tier classifier architecture. A system configured to practice the method can receive, from a first classifier, a first voice activity indicator detected in a first modality for a human subject. Then, the system can receive, from a second classifier, a second voice activity indicator detected in a second modality for the human subject, wherein the first voice activity indicator and the second voice activity indicators are based on the human subject at a same time, and wherein the first modality and the second modality are different. The system can concatenate, via a third classifier, the first voice activity indicator and the second voice activity indicator with original features of the human subject, to yield a classifier output, and determine voice activity based on the classifier output.

We claim: 
     
         1 . A method comprising:
 receiving, from a first classifier, a first modality output detected from a first input from a human subject during a first time period;   receiving, from a second classifier, a second modality output detected from second input from the human subject during a second time period, wherein the first time period and the second time period differ;   combining first classifier output from the first classifier with second classifier output from the second classifier to yield classifier group output; and   determining voice activity based on the classifier group output.   
     
     
         2 . The method of  claim 1 , wherein an acoustic feature associated with the first input comprises one of Mel-frequency cepstral coefficients, a first derivative of Mel-frequency cepstral coefficients, a second derivative of Mel-frequency cepstral coefficients, and acoustic energy. 
     
     
         3 . The method of  claim 1 , wherein a visual feature associated with the second input comprises one of a parametric feature, an appearance-based feature, dimensions of a mouth region, image region intensity, discrete cosine transformations, image saturation, image brightness, image texture, video motion, head movement, one of movement of a specific item relative to other items and movement of the specific item relative to a background, micro-expressions on a person's face, extremity of expression, detected lighting changes, head angle relative to a camera, and local binary patterns. 
     
     
         4 . The method of  claim 1 , wherein original features associated with the human subject are associated with a video frame. 
     
     
         5 . The method of  claim 1 , wherein the combining of the first classifier output from the first classifier with the second classifier output from the second classifier further comprises:
 performing late fusion of original features associated with the human subject using weighted majority voting.   
     
     
         6 . The method of  claim 1 , wherein the first input comprises voice input. 
     
     
         7 . The method of  claim 1 , wherein the second input comprises visual input. 
     
     
         8 . A system comprising:
 a processor; and   a computer-readable medium having instructions which, when executed by the processor, cause the processor to perform operations comprising:
 receiving, from a first classifier, a first modality output detected from a first input from a human subject during a first time period; 
 receiving, from a second classifier, a second modality output detected from second input from the human subject during a second time period, wherein the first time period and the second time period differ; 
 combining first classifier output from the first classifier with second classifier output from the second classifier to yield classifier group output; and 
 determining voice activity based on the classifier group output. 
   
     
     
         9 . The system of  claim 8 , wherein an acoustic feature associated with the first input comprises one of Mel-frequency cepstral coefficients, a first derivative of Mel-frequency cepstral coefficients, a second derivative of Mel-frequency cepstral coefficients, and acoustic energy. 
     
     
         10 . The system of  claim 8 , wherein a visual feature associated with the second input comprises one of a parametric feature, an appearance-based feature, dimensions of a mouth region, image region intensity, discrete cosine transformations, image saturation, image brightness, image texture, video motion, head movement, one of movement of a specific item relative to other items and movement of the specific item relative to a background, micro-expressions on a person's face, extremity of expression, detected lighting changes, head angle relative to a camera, and local binary patterns. 
     
     
         11 . The system of  claim 8 , wherein original features associated with the human subject are associated with a video frame. 
     
     
         12 . The system of  claim 8 , wherein the combining of the first classifier output from the first classifier with the second classifier output from the second classifier further comprises:
 performing late fusion of original features associated with the human subject using weighted majority voting.   
     
     
         13 . The system of  claim 8 , wherein the first input comprises voice input. 
     
     
         14 . The system of  claim 8 , wherein the second input comprises visual input. 
     
     
         15 . A computer-readable storage device storing instructions which, when executed by a computing device, cause the computing device to perform operations comprising:
 receiving, from a first classifier, a first modality output detected from a first input from a human subject during a first time period;   receiving, from a second classifier, a second modality output detected from second input from the human subject during a second time period, wherein the first time period and the second time period differ;   combining first classifier output from the first classifier with second classifier output from the second classifier to yield classifier group output; and   determining voice activity based on the classifier group output.   
     
     
         16 . The computer-readable storage device of  claim 15 , wherein an acoustic feature associated with the first input comprises one of Mel-frequency cepstral coefficients, a first derivative of Mel-frequency cepstral coefficients, a second derivative of Mel-frequency cepstral coefficients, and acoustic energy. 
     
     
         17 . The computer-readable storage device of  claim 15 , wherein a visual feature associated with the second input comprises one of a parametric feature, an appearance-based feature, dimensions of a mouth region, image region intensity, discrete cosine transformations, image saturation, image brightness, image texture, video motion, head movement, one of movement of a specific item relative to other items and movement of the specific item relative to a background, micro-expressions on a person's face, extremity of expression, detected lighting changes, head angle relative to a camera, and local binary patterns. 
     
     
         18 . The computer-readable storage device of  claim 15 , wherein original features associated with the human subject are associated with a video frame. 
     
     
         19 . The computer-readable storage device of  claim 15 , wherein the combining of the first classifier output from the first classifier with the second classifier output from the second classifier further comprises:
 performing late fusion of original features associated with the human subject using weighted majority voting.   
     
     
         20 . The computer-readable storage device of  claim 15 , wherein the first input comprises voice input or wherein the second input comprises visual input.