Engaging in human-based social interaction with members of a group using a persistent companion device

Abstract

A persistent companion robot supports both one-on-one interaction with a human and group interaction with more than one human. The interaction can be directed to a human in detectable proximity, such as a human that is near to the robot, one that is further away from the robot, or any combination of near and far humans. The interaction incorporates multi-modal human input detection (e.g., seeing, hearing, tactile) with multi-modal expression (e.g., movement, speech, non-speech sound, lighting, electronic imagery, and the like.

What is claimed is: 
     
         1 . A method comprising:
 receiving audio information comprising one or more audio signals at a social robot via one or more microphones;   applying a beam forming algorithm to the audio information to:
 isolate an audio source for at least one of the audio signals; and 
 determine a spatial location of the isolated audio source; and 
   orienting a portion of the social robot based, at least in part, on the determined spatial location.   
     
     
         2 . The method of  claim 1 , wherein the one or more microphones comprises an array of microphones disposed to facilitate detecting differences in sound time-of arrival of the audio information. 
     
     
         3 . The method of  claim 1 , wherein the beam forming algorithm comprises a detection algorithm for discriminating between speech and non-speech. 
     
     
         4 . The method of  claim 3 , further comprising utilizing the detection algorithm to identify an isolated audio source as a speaker. 
     
     
         5 . The method of  claim 1 , wherein the beam forming algorithm applies information from a 3D sensor of the robot to determine the spatial location. 
     
     
         6 . The method of  claim 5 , further comprising utilizing the 3D sensor to identify a direction from the social robot to a head of the speaker and transmitting the direction to an automatic speech recognizer (ASR) module of the social robot. 
     
     
         7 . The method of  claim 1 , wherein the social robot comprises a plurality of separate speech recognizers each of which is for processing an audio signal from a distinct isolated audio source determined via the beam forming algorithm. 
     
     
         8 . The method of  claim 7 , wherein each of the speech recognizers is configured to recognize a hot phrase. 
     
     
         9 . The method of  claim 8 , wherein the orienting the portion of the social robot is further based, at least in part, upon, the recognition of the hot phrase. 
     
     
         10 . The method of  claim 1 , further comprising integrating information indicative of the isolated audio source from a vision system of the social robot to determine the spatial location. 
     
     
         11 . The method of  claim 1 , further comprising integrating information indicative of the isolated audio source from an attention system of the social robot to determine the spatial location. 
     
     
         12 . A method comprising:
 receiving and storing at a social robot near-field originating data indicative of an attribute of at least one person;   receiving and storing at the social robot far-field originating data indicative of an attribute of at least one other person;   identifying at least one of the people based, at least in part, on at least one of the near-field originating data and the far-field originating data; and   engaging in a social interaction with the at least one identified person via a mode of interaction of the social robot selected from the list consisting of speech, animation and movement, wherein the social interaction is modulated based on whether the identified person is in the near-field or the far-field.   
     
     
         13 . The method of  claim 12 , further comprising tracking the at least one identified person with a beam formed from audio data received by at least one microphone of the social robot. 
     
     
         14 . The method of  claim 12 , wherein the near-field originating data is captured via one or more touch sensitive sensors of the social robot. 
     
     
         15 . The method of  claim 12 , further comprising processing at least one of the near-field originating data and the far-field originating data using a grid-based particle filter model to determine a physical state of at least one of the persons. 
     
     
         16 . The method of  claim 12 , further comprising disengaging with the at least one identified person and engaging with the at least one other person based, at least in part, upon an indication by the at least one other person of a desire to speak. 
     
     
         17 . The method of  claim 16 , wherein the indication is derived from at least one of a non-verbal and a paralinguistic social cue. 
     
     
         18 . The method of  claim 12 , wherein a mode of engaging in the social interaction is based, at least in part, on an attribute of the person interacting with the social robot, the attribute selected from the list consisting of age, emotional state, gender, posture, gaze direction and degree of fatigue. 
     
     
         19 . The method of  claim 12 , wherein engaging in a social interaction comprises engaging in an interactive story telling exercise that comprises:
 receiving audio data comprising a reading of a story; and   directing audio produced from the received audio data to at least one of the at least one person and the at least one other person.   
     
     
         20 . The method of  claim 19 , wherein engaging in the interactive story telling exercise further comprises applying a preset filter to the audio data. 
     
     
         21 . The method of  claim 19 , wherein engaging in the interactive story telling exercise further comprises at least one of displaying on a display device of the social robot visual indicia of an attribute of the audio data and movement of at least one moveable segment of the social robot. 
     
     
         22 . The method of  claim 12 , further comprising utilizing at least one of a heuristic proposal distribution and a heuristic transition model to capture a state of at least one of the at least one person and the at least one other person. 
     
     
         23 . The method of  claim 12 , further comprising storing information describing the appearance and disappearance from persons within both the near-field and far-field. 
     
     
         24 . The method of  claim 12 , wherein engaging in a social interaction comprises moderating a meeting. 
     
     
         25 . A method comprising:
 isolating audio sources for a portion of the plurality of audio signals by applying a beam forming algorithm to the audio information;   configuring distinct audio source beams for a portion of the isolated audio sources, each beam indicating a radial direction relative to the social robot of the portion of the isolated audio sources;   processing the audio signals from the portion of isolated audio sources with audio source beam-specific instances of a speech recognizer algorithm executing on the social robot, the speech recognizer being adapted to detect an attention keyword in the audio signal; and   orienting a movable portion of the social robot based, at least in part, on the determined radial direction of the beam associated with the speech recognizer that detects the attention keyword.   
     
     
         26 . The method of  claim 25 , wherein configuring distinct audio source beams is based on microphone orientation information provided by a motion controller of the social robot. 
     
     
         27 . The method of  claim 25 , further comprising updating at least one of the distinct audio source beams based on audio source location information provided by a 3D sensor of the social robot.