System and method for intelligent initiation of a man-machine dialogue based on multi-modal sensory inputs

Abstract

The present teaching relates to method, system, medium, and implementations for enabling communication with a user. Information representing surrounding of a user to be engaged in a new dialogue is received via the communication platform, wherein the information is acquired from a scene in which the user is present and captures characteristics of the user and the scene. Relevant features are extracted from the information. A state of the user is estimated based on the relevant features, and a dialogue context surrounding the scene is determined based on the relevant features. A topic for the new dialogue is determined based on the user, and a feedback is generated to initiate the new dialogue with the user based on the topic, the state of the user, and the dialogue context.

We claim: 
     
         1 . A method, implemented on a machine having at least one processor, storage, and a communication platform for initiating a dialogue with a user, comprising:
 receiving, by a dialogue agent via the communication platform, information capturing a user and a surrounding scene, wherein the user is to be engaged in a new dialogue in the scene;   estimating a state of the user based on features extracted from the information;   selecting, from a plurality of programs, a program on a first topic for the user, wherein the program on the first topic corresponds to an intended dialogue;   determining whether the intended dialogue on the first topic associated with the first program is to be initiated given the state of the user;   initiating the intended dialogue with the user as the new dialogue, if the intended dialogue is appropriate according to the state of the user; and   initiating, if the intended dialogue is currently inappropriate for the user, an alternative dialogue on an alternative topic with the user as the new dialogue, wherein the alternative topic is determined based on the state of the user, observed features of the user, and a dialogue context of the scene determined based on the relevant features.   
     
     
         2 . The method of  claim 1 , wherein
 when the state of the user indicates that the user is likely able to engage in the intended dialogue on the first topic, the intended dialogue is considered appropriate in the state of the user; and   when the state of the user indicates that the user likely is not able to engage in the intended dialogue on the first topic, the intended dialogue is considered not appropriate in the state of the user.   
     
     
         3 . The method of  claim 1 , wherein the information includes multimodal sensor data in at least one of audio, visual, textual, and haptic modalities, where the audio sensor data record acoustic sound from the scene and/or a speech from the user. 
     
     
         4 . The method of  claim 1 , wherein the state of the user characterizes at least one of:
 an appearance of the user observed in the scene;   an expression of the user estimated based on the information acquired from the scene;   one or more emotions of the user inferred based on the expression of the user; and   an intent of the user inferred based on at least one of the expression and the one or more emotions.   
     
     
         5 . The method of  claim 1 , wherein the dialogue context includes at least one of:
 at least one object present in the scene and a characterization thereof;   an estimated classification of the scene;   a characterization of the scene; and   a sound heard from the environment of the scene.   
     
     
         6 . The method of  claim 1 , wherein the first topic is one of
 a subject matter related to the program that the user previously signed up;   a subject matter dynamically determined; and   a combination thereof.   
     
     
         7 . The method of  claim 1 , wherein the new dialogue is initiated via at least one of speech and visual means. 
     
     
         8 . Machine readable and non-transitory medium having information recorded thereon for initiating a dialogue with a user, wherein the information, once read by the machine, causes the machine to perform the following steps:
 receiving, by a dialogue agent via the communication platform, information capturing a user and a surrounding scene, wherein the user is to be engaged in a new dialogue in the scene;   estimating a state of the user based on features extracted from the information;   selecting, from a plurality of programs, a program on a first topic for the user, wherein the program on the first topic corresponds to an intended dialogue;   determining whether the intended dialogue on the first topic associated with the first program is to be initiated given the state of the user;   initiating the intended dialogue with the user as the new dialogue, if the intended dialogue is appropriate according to the state of the user; and   initiating, if the intended dialogue is currently inappropriate for the user, an alternative dialogue on an alternative topic with the user as the new dialogue, wherein the alternative topic is determined based on the state of the user, observed features of the user, and a dialogue context of the scene determined based on the relevant features.   
     
     
         9 . The medium of  claim 8 , wherein
 when the state of the user indicates that the user is likely able to engage in the intended dialogue on the first topic, the intended dialogue is considered appropriate in the state of the user; and   when the state of the user indicates that the user likely is not able to engage in the intended dialogue on the first topic, the intended dialogue is considered not appropriate in the state of the user.   
     
     
         10 . The medium of  claim 8 , wherein the information includes multimodal sensor data in at least one of audio, visual, textual, and haptic modalities, where the audio sensor data record acoustic sound from the scene and/or a speech from the user. 
     
     
         11 . The medium of  claim 8 , wherein the state of the user characterizes at least one of:
 an appearance of the user observed in the scene;   an expression of the user estimated based on the information acquired from the scene;   one or more emotions of the user inferred based on the expression of the user; and   an intent of the user inferred based on at least one of the expression and the one or more emotions.   
     
     
         12 . The medium of  claim 8 , wherein the dialogue context includes at least one of:
 at least one object present in the scene and a characterization thereof;   an estimated classification of the scene;   a characterization of the scene; and   a sound heard from the environment of the scene.   
     
     
         13 . The medium of  claim 8 , wherein the first topic is one of
 a subject matter related to the program that the user previously signed up;   a subject matter dynamically determined; and   a combination thereof.   
     
     
         14 . The medium of  claim 8 , wherein the new dialogue is initiated via at least one of speech and visual means. 
     
     
         15 . A system for initiating a dialogue with a user, comprising:
 a multimodal data analysis unit implemented on a processor and configured for
 receiving, by a dialogue agent via the communication platform, information capturing a user and a surrounding scene, wherein the user is to be engaged in a new dialogue in the scene, and 
 extracting features from the information; 
   a user state estimator implemented on a processor and configured for estimating a state of the user based on the features extracted from the information;   a dialogue controller implemented on a processor and configured for
 selecting, from a plurality of programs, a program on a first topic for the user, wherein the program on the first topic corresponds to an intended dialogue, 
 determining whether the intended dialogue on the first topic associated with the first program is to be initiated given the state of the user, 
 initiating the intended dialogue with the user as the new dialogue, if the intended dialogue is appropriate according to the state of the user, and 
 initiating, if the intended dialogue is currently inappropriate for the user, an alternative dialogue on an alternative topic with the user as the new dialogue, wherein the alternative topic is determined based on the state of the user, observed features of the user, and a dialogue context of the scene determined based on the relevant features. 
   
     
     
         16 . The system of  claim 15 , wherein
 when the state of the user indicates that the user is likely able to engage in the intended dialogue on the first topic, the intended dialogue is considered appropriate in the state of the user; and   when the state of the user indicates that the user likely is not able to engage in the intended dialogue on the first topic, the intended dialogue is considered not appropriate in the state of the user.   
     
     
         17 . The system of  claim 15 , wherein the information includes multimodal sensor data in at least one of audio, visual, textual, and haptic modalities, where the audio sensor data record acoustic sound from the scene and/or a speech from the user. 
     
     
         18 . The system of  claim 15 , wherein the state of the user characterizes at least one of:
 an appearance of the user observed in the scene;   an expression of the user estimated based on the information acquired from the scene;   one or more emotions of the user inferred based on the expression of the user; and   an intent of the user inferred based on at least one of the expression and the one or more emotions.   
     
     
         19 . The system of  claim 15 , wherein the dialogue context includes at least one of:
 at least one object present in the scene and a characterization thereof;   an estimated classification of the scene;   a characterization of the scene; and   a sound heard from the environment of the scene.   
     
     
         20 . The system of  claim 15 , wherein the first topic is one of
 a subject matter related to the program that the user previously signed up;   a subject matter dynamically determined; and   a combination thereof.   
     
     
         21 . The system of  claim 15 , wherein the new dialogue is initiated via at least one of speech and visual means.