Augmented multi-tier classifier for multi-modal voice activity detection

Abstract

Voice activity in a media signal is detected in an augmented, multi-tier classifier architecture. For instance, a first voice activity indicator, detected in a first modality for a human subject, is received from a first classifier. Then, the system can receive, from a second classifier, a second voice activity indicator detected in a second modality for the human subject, wherein the first voice activity indicator and the second voice activity indicators are based on the human subject at a same time, and wherein the first modality and the second modality are different. The system then concatenates, via a third classifier, the first voice activity indicator and the second voice activity indicator with original features of the human subject, to yield a classifier output, and determine voice activity based on the classifier output.

What is claimed is: 
     
         1 . A method, comprising:
 generating, by a system comprising a processor, via a group of first tier classifiers using respective feature vectors representing features from multiple modalities of data captured from inputs comprising audio inputs and visual inputs, respective first decision outputs of first tier classifiers of the group of first tier classifiers related to classifying the inputs as human voice activity; and   determining, by the system, via a second tier classifier using the respective feature vectors and the respective first decision outputs of the first tier classifiers, a second decision output related to classifying the inputs as the human voice activity.   
     
     
         2 . The method of  claim 1 , further comprising determining, by the system, whether the inputs comprise the human voice activity based on the second decision output. 
     
     
         3 . The method of  claim 1 , further comprising combining, by the system, via the second tier classifier, the respective feature vectors and the respective first decision outputs. 
     
     
         4 . The method of  claim 1 , further comprising concatenating, by the system, via the second tier classifier, the respective feature vectors and the respective first decision outputs. 
     
     
         5 . The method of  claim 1 , wherein the features comprise acoustic features comprising at least one of Mel-frequency cepstral coefficients, a first derivative of Mel-frequency cepstral coefficients, a second derivative of Mel-frequency cepstral coefficients, or an acoustic energy. 
     
     
         6 . The method of  claim 1 , wherein the features comprise visual features comprising at least one of a parametric feature, an appearance-based feature, a dimension of a mouth region, image region intensity, a discrete cosine transformation, an image saturation, an image brightness, an image texture, a video motion, a head movement, a movement of a specific item relative to other items, a movement of the specific item relative to a background, a micro-expression on a face, an extremity of expression, a detected lighting change, a head angle relative to a camera, or a local binary pattern. 
     
     
         7 . The method of  claim 1 , wherein the first tier classifiers are respectively trained for different types of the features. 
     
     
         8 . A system, comprising:
 a processor; and   a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations, comprising:
 generating, via first tier classifiers using respective feature vectors representing features from multiple modalities of data captured from inputs comprising audio inputs and visual inputs, respective first classifier outputs of the first tier classifiers related to classifying the inputs as human voice activity; and 
 determining, via a second tier classifier using the respective feature vectors and the respective first classifier outputs of the first tier classifiers, a second classifier output related to classifying the inputs as the human voice activity. 
   
     
     
         9 . The system of  claim 8 , wherein the operations further comprise determining whether the inputs comprise the human voice activity based on the second classifier output. 
     
     
         10 . The system of  claim 8 , wherein the respective first classifier outputs comprise output vectors. 
     
     
         11 . The system of  claim 10 , wherein the operations further comprise, concatenating, via the second tier classifier, the respective feature vectors and the output vectors. 
     
     
         12 . The system of  claim 8 , wherein the features comprise acoustic features comprising at least one of Mel-frequency cepstral coefficients, a first derivative of Mel-frequency cepstral coefficients, a second derivative of Mel-frequency cepstral coefficients, or acoustic energy. 
     
     
         13 . The system of  claim 8 , wherein the features comprise visual features comprising at least one of a parametric feature, an appearance-based feature, dimensions of a mouth region, image region intensity, discrete cosine transformations, image saturation, image brightness, image texture, video motion, head movement, movement of a specific item relative to other items, movement of the specific item relative to a background, micro-expressions on a person's face, extremity of expression, detected lighting changes, head angle relative to a camera, or local binary patterns. 
     
     
         14 . The system of  claim 8 , wherein the first tier classifiers are respectively trained for different types of the features. 
     
     
         15 . A non-transitory machine-readable medium, comprising executable instructions that, when executed by a processor, facilitate performance of operations, comprising:
 generating, via first tier classifiers using respective feature vectors representing features from multiple modalities of data captured from inputs comprising audio inputs and visual inputs, respective first classifier outputs of the first tier classifiers related to classifying the inputs as human voice activity; and   determining, via a second tier classifier using the respective feature vectors and the respective first classifier outputs of the first tier classifiers, a second classifier output related to classifying the inputs as the human voice activity.   
     
     
         16 . The non-transitory machine-readable medium of  claim 15 , wherein the operations further comprise determining whether the inputs comprise the human voice activity based on the second classifier output. 
     
     
         17 . The non-transitory machine-readable medium of  claim 15 , wherein the operations further comprise combining, via the second tier classifier, the respective feature vectors and the respective first classifier outputs. 
     
     
         18 . The non-transitory machine-readable medium of  claim 15 , wherein the features comprise acoustic features comprising at least one of Mel-frequency cepstral coefficients, a first derivative of Mel-frequency cepstral coefficients, a second derivative of Mel-frequency cepstral coefficients, or acoustic energy. 
     
     
         19 . The non-transitory machine-readable medium of  claim 15 , wherein the features comprise visual features comprising at least one of a parametric feature, an appearance-based feature, dimensions of a mouth region, image region intensity, discrete cosine transformations, image saturation, image brightness, image texture, video motion, head movement, movement of a specific item relative to other items, movement of the specific item relative to a background, micro-expressions on a person's face, extremity of expression, detected lighting changes, head angle relative to a camera, or local binary patterns. 
     
     
         20 . The non-transitory machine-readable medium of  claim 15 , wherein the first tier classifiers are respectively trained for different types of the features.