Multi-modal sensor fusion for content identification in applications of human-machine interfaces

Abstract

Interactions with virtual systems may be difficult when users inadvertently fail to provide sufficient information to proceed with their requests. Certain types of inputs, such as auditory inputs, may lack sufficient information to properly provide a response to the user. Additional information, such as image data, may enable user gestures or poses to supplement the auditory inputs to enable response generation without requesting additional information from users.

What is claimed is: 
     
         1 . A processor comprising:
 one or more circuits to:
 determine, from an auditory query of a user, an intent; 
 determine, from visual information associated with the user, one or more items in a vicinity of the user; and 
 generate, based on a relationship between the intent and the one or more items, a response to the auditory query. 
   
     
     
         2 . The processor of  claim 1 , wherein the one or more circuits are further to:
 determine the auditory query provides insufficient information to identify the one or more items; and   identify, based at least in part on the visual information and the auditory query, the one or more items.   
     
     
         3 . The processor of  claim 1 , wherein the one or more circuits are further to:
 determine a user pose from the visual information, wherein the one or more items are determined, based at least one an identification criteria for the user pose.   
     
     
         4 . The processor of  claim 3 , wherein the one or more circuits are further to:
 generate one or more bounding shapes corresponding to the one or more items;   identify one or more portions of the visual information associated with the user; and   determine one or more overlapping regions between the one or more portions and the one or more bounding shapes.   
     
     
         5 . The processor of  claim 3 , wherein the one or more circuits are further to:
 identify a first point of the user and a second point of the user;   generate a virtual ray projecting through the first point and the second point; and   determine the virtual ray intersects a region of interest associated with the one or more items.   
     
     
         6 . The processor of  claim 1 , wherein the one or more circuits are further to:
 extract, from the auditory query, the intent and an entity; and   determine, based at least in part on an identification criteria, that the entity is not provided in the auditory query.   
     
     
         7 . The processor of  claim 1 , wherein the auditory query is analyzed using at least one of automatic speech recognition (ASR), natural language processing (NLP), natural language understanding (NLU), or text-to-speech. 
     
     
         8 . The processor of  claim 1 , wherein the one or more circuits are further to:
 fill an intent slot using the intent extracted from the auditory query;   fill an entity slot using information associated with the one or more items; and   process the intent slot and the entity slot to generate the response to the auditory query.   
     
     
         9 . The processor of  claim 1 , wherein the visual information is three-dimensional sensor data and the one or more items are identified using a point cloud. 
     
     
         10 . A method, comprising:
 determining an intent from an auditory query provided by a user;   determining, based at least on visual information corresponding to the user, one or more items associated with the user; and   generating, based on a relationship between the intent and the one or more items, a response to the auditory query.   
     
     
         11 . The method of  claim 10 , wherein the one or more items are depicted in or represented by the visual information. 
     
     
         12 . The method of  claim 10 , further comprising:
 determining the auditory query provides insufficient information to identify the one or more items; and   identifying, based at least in part on the visual information and the auditory query, the one or more items.   
     
     
         13 . The method of  claim 10 , further comprising:
 determining a user pose from the visual information, wherein the one or more items are determined, based at least one an identification criteria for the user pose.   
     
     
         14 . The method of  claim 13 , further comprising:
 generating one or more bounding shapes corresponding to the one or more items;   identifying one or more regions of the visual information corresponding to the user; and   determining one or more overlapping regions between the one or more regions and the one or more bounding shapes.   
     
     
         15 . The method of  claim 13 , further comprising:
 identifying a first point of the user and a second point of the user;   generating a virtual ray projecting through the first point and the second point; and   determining the virtual ray intersects a region of interest associated with the one or more items.   
     
     
         16 . A system, comprising:
 an imaging device;   an audio device;   at least one central processing unit; and   at least one graphics processing unit,   wherein the system is to:
 determine an intent from an auditory query; 
 determine, based at least on visual information of a user associated with the auditory query, one or more items; and 
 generate, based on a relationship between the intent and the one or more items, a response to the auditory query. 
   
     
     
         17 . The system of  claim 16 , wherein the system is further to:
 determine, from the visual information, a pose of the user; and   identify a query entity, based on the pose of the user.   
     
     
         18 . The system of  claim 17 , wherein the pose corresponds to a holding pose in which the user is touching the one or more items. 
     
     
         19 . The system of  claim 17 , wherein the pose corresponds to a pointing pose in which one or more appendages of the user are pointing toward the one or more items. 
     
     
         20 . The system of  claim 16 , wherein the system is further to:
 determine the auditory query provides insufficient information to identify the one or more items; and   identifying, based at least in part on the visual information and the audio query, the one or more items.