System and method for disambiguating a source of sound based on detected lip movement

Abstract

The present teaching relates to method, system, medium, and implementations for detecting a source of speech sound in a dialogue. A visual signal acquired from a dialogue scene is first received, where the visual signal captures a person present in the dialogue scene. A human lip associated with the person is detected from the visual signal and tracked to detect whether lip movement is observed. If lip movement is detected, a first candidate source of sound is generated corresponding to an area in the dialogue scene where the lip movement occurred.

We claim: 
     
         1 . A method implemented on at least one machine including at least one processor, memory, and communication platform capable of connecting to a network for detecting a source of speech sound in a dialogue, the method comprising:
 receiving sensor data acquired by a plurality of sensors from a dialogue scene, wherein the sensor data capture information about the dialogue scene having one or more people present including a person currently engaged in a human machine dialogue;   estimating at least one candidate source of sound based on the sensor data, wherein each of the at least one candidate source of sound is characterized by one or more parameters;   obtaining user information related to the person, wherein the user information characterizes a current state of the person, including a location of the person most recently tracked; and   identifying one or more candidate sources of speech sound from the at least one candidate source of sound via filtering based on the location of the person and the one or more parameters associated with each of the at least one candidate source of sound.   
     
     
         2 . The method of  claim 1 , wherein the plurality of sensors include
 at least one visual sensor for capturing visual information of the dialogue scene; or   at least one audio sensor for capturing audio information in the dialogue scene, wherein   the plurality of sensors are distributed in the dialogue scene.   
     
     
         3 . The method of  claim 2 , wherein the step of estimating at least one candidate source of sound comprises:
 receiving audio information from the at least one audio sensor; and   for each of the at least one audio sensor,
 estimating a source of sound associated with the audio sensor if sound is detected based on the audio information from the audio sensor, and 
 determining that the estimated source of sound qualifies as a candidate source of sound via confirmation, wherein 
 the one or more parameters characterizing the candidate source of sound includes a location determined based on that of the audio sensor. 
   
     
     
         4 . The method of  claim 3 , wherein the confirmation includes at least one of:
 ensuring that the sound detected is not faint;   at least one of the one or more people is present near where the audio sensor is deployed; and   lip movement of the at least one or more people is detected.   
     
     
         5 . The method of  claim 2 , wherein the step of estimating at least one candidate source of sound comprises:
 receiving visual information from the at least one visual sensor;   for each of the at least one visual sensor,
 estimating a source of sound associated with the visual sensor when lip movement is detected from any of the one or more people based on the visual information from the visual sensor, and 
 determining that the estimated source of sound qualifies as a candidate source of sound through an acoustic confirmation, wherein 
 the one or more parameters characterizing the candidate source of sound includes a location determined based on that of the audio sensor. 
   
     
     
         6 . The method of  claim 5 , wherein the acoustic confirmation ensures that the lip movement is associated with a detected sound. 
     
     
         7 . The method of  claim 2 , wherein the one or more parameters characterizing each of the at least one candidate source of sound indicate:
 with respect to a candidate source of sound identified based on the audio information from an audio sensor,
 location information associated with the audio sensor, and 
 whether the audio information represents human speech; and 
   with respect to a candidate source of sound detected based on the visual information from a visual sensor,
 location information associated with the visual sensor, and 
 whether the lip movement detected from the visual information from the visual sensor corresponds to speech lip movement. 
   
     
     
         8 . Machine readable medium having information recorded thereon for detecting a source of speech sound in a dialogue, wherein the information, when read by the machine, causes the machine to perform the following steps:
 receiving sensor data acquired by a plurality of sensors from a dialogue scene, wherein the sensor data capture information about the dialogue scene having one or more people present including a person currently engaged in a human machine dialogue;   estimating at least one candidate source of sound based on the sensor data, wherein each of the at least one candidate source of sound is characterized by one or more parameters;   obtaining user information related to the person, wherein the user information characterizes a current state of the person, including a location of the person most recently tracked; and   identifying one or more candidate sources of speech sound from the at least one candidate source of sound via filtering based on the location of the person and the one or more parameters associated with each of the at least one candidate source of sound.   
     
     
         9 . The medium of  claim 8 , wherein the plurality of sensors include
 at least one visual sensor for capturing visual information of the dialogue scene; or   at least one audio sensor for capturing audio information in the dialogue scene, wherein   the plurality of sensors are distributed in the dialogue scene.   
     
     
         10 . The medium of  claim 9 , wherein the step of estimating at least one candidate source of sound comprises:
 receiving audio information from the at least one audio sensor; and   for each of the at least one audio sensor,
 estimating a source of sound associated with the audio sensor if sound is detected based on the audio information from the audio sensor, and 
 determining that the estimated source of sound qualifies as a candidate source of sound via confirmation, wherein 
 the one or more parameters characterizing the candidate source of sound includes a location determined based on that of the audio sensor. 
   
     
     
         11 . The medium of  claim 10 , wherein the confirmation includes at least one of:
 ensuring that the sound detected is not faint;   at least one of the one or more people is present near where the audio sensor is deployed; and   lip movement of the at least one or more people is detected.   
     
     
         12 . The medium of  claim 9 , wherein the step of estimating at least one candidate source of sound comprises:
 receiving visual information from the at least one visual sensor;   for each of the at least one visual sensor,
 estimating a source of sound associated with the visual sensor when lip movement is detected from any of the one or more people based on the visual information from the visual sensor, and 
 determining that the estimated source of sound qualifies as a candidate source of sound through an acoustic confirmation, wherein 
 the one or more parameters characterizing the candidate source of sound includes a location determined based on that of the audio sensor. 
   
     
     
         13 . The medium of  claim 12 , wherein the acoustic confirmation ensures that the lip movement is associated with a detected sound. 
     
     
         14 . The medium of  claim 9 , wherein the one or more parameters characterizing each of the at least one candidate source of sound indicate:
 with respect to a candidate source of sound identified based on the audio information from an audio sensor,
 location information associated with the audio sensor, and 
 the audio information represents human speech; and 
   with respect to a candidate source of sound detected based on the visual information from a visual sensor,
 location information associated with the visual sensor, and 
 whether the lip movement detected from the visual information from the visual sensor corresponds to speech lip movement. 
   
     
     
         15 . A system for detecting a source of speech sound in a dialogue, comprising:
 a plurality of sound source estimators implemented using a processor and configured for
 receiving sensor data acquired by a plurality of sensors from a dialogue scene, wherein the sensor data capture information about the dialogue scene having one or more people present including a person currently engaged in a human machine dialogue, 
 estimating at least one candidate source of sound based on the sensor data, wherein each of the at least one candidate source of sound is characterized by one or more parameters, and 
 obtaining user information related to the person, wherein the user information characterizes a current state of the person, including a location of the person most recently tracked; and 
   a sound source disambiguation unit implemented using a processor and configured for identifying one or more candidate sources of speech sound from the at least one candidate source of sound via filtering based on the location of the person and the one or more parameters associated with each of the at least one candidate source of sound.   
     
     
         16 . The system of  claim 15 , wherein the plurality of sensors include:
 at least one visual sensor for capturing visual information of the dialogue scene; or   at least one audio sensor for capturing audio information in the dialogue scene, wherein   the plurality of sensors are distributed in the dialogue scene.   
     
     
         17 . The system of  claim 16 , wherein the plurality of sound source estimators include an audio based sound source estimator, implemented using a processor and configured for estimating at least one candidate source of sound, comprising:
 receiving audio information from the at least one audio sensor; and   for each of the at least one audio sensor,
 estimating a source of sound associated with the audio sensor if sound is detected based on the audio information from the audio sensor, and 
 determining that the estimated source of sound qualifies as a candidate source of sound via confirmation, wherein 
 the one or more parameters characterizing the candidate source of sound includes a location determined based on that of the audio sensor. 
   
     
     
         18 . The system of  claim 17 , wherein the confirmation includes at least one of:
 ensuring that the sound detected is not faint;   at least one of the one or more people is present near where the audio sensor is deployed; and   lip movement of the at least one or more people is detected.   
     
     
         19 . The system of  claim 16 , wherein the plurality of sound source estimators include a visual based sound source estimator, implemented using a processor and configured for estimating at least one candidate source of sound, comprising:
 receiving visual information from the at least one visual sensor;   for each of the at least one visual sensor,
 estimating a source of sound associated with the visual sensor when lip movement is detected from any of the one or more people based on the visual information from the visual sensor, and 
 determining that the estimated source of sound qualifies as a candidate source of sound through an acoustic confirmation, wherein 
 the one or more parameters characterizing the candidate source of sound includes a location determined based on that of the audio sensor. 
   
     
     
         20 . The system of  claim 19 , wherein the acoustic confirmation ensures that the lip movement is associated with a detected sound. 
     
     
         21 . The system of  claim 16 , wherein the one or more parameters characterizing each of the at least one candidate source of sound indicate:
 with respect to a candidate source of sound identified based on the audio information from an audio sensor,
 location information associated with the audio sensor, and 
 whether the audio information represents human speech; and 
   with respect to a candidate source of sound detected based on the visual information from a visual sensor,
 location information associated with the visual sensor, and 
 whether the lip movement detected from the visual information from the visual sensor corresponds to speech lip movement.