Semantically tagged virtual and physical objects

Abstract

A head mounted display device is provided that includes a display device, a camera device, an input device, and a processor. The processor is configured to store a database of physical objects and virtual objects that have been associated with one or more semantic tags. The processor is further configured to receive a natural language input from a user via the input device and perform semantic processing on the natural language input to determine a user specified operation and identify one or more semantic tags indicated by the natural language input. The processor is further configured to select a target virtual object and a target physical object based on the identified one or more semantic tags, perform the determined user specified operation on the target virtual object based on the target physical object, and display the target virtual object at a physical location associated with the target physical object.

1 . A head mounted display device comprising:
 a display device configured to display virtual objects at locations in a physical environment;   a camera device configured to capture images of the physical environment;   an input device configured to receive a user input; and   a processor configured to:
 store a database of physical objects and virtual objects that have been associated with one or more semantic tags; 
 receive a natural language input from a user via the input device; 
 perform semantic processing on the natural language input to determine a user specified operation and identify one or more semantic tags indicated by the natural language input; 
 select a target virtual object and a target physical object from the physical objects and virtual objects in the database based on the identified one or more semantic tags; 
 perform the determined user specified operation on the target virtual object based on the target physical object; and 
 display the target virtual object at a physical location associated with the target physical object. 
   
     
     
         2 . The head mounted display device of  claim 1 , wherein the physical objects are selected from the group consisting of a room, a wall, a room furnishing, a table, a chair, a surface, physical location, and a person. 
     
     
         3 . The head mounted display device of  claim 1 , wherein the physical location of the target physical object is outside of a field of view of the camera device of the head mounted display device. 
     
     
         4 . The head mounted display device of  claim 1 , wherein the natural language input is a voice input received via the input device. 
     
     
         5 . The head mounted display device of  claim 1 , wherein the determined user specified operation is a move operation, and to perform the determined user specified operation the processor is further configured to:
 update a location of the target virtual object in the physical environment based on the physical location associated with the target physical object; and   display the target virtual object at the updated location via the display device.   
     
     
         6 . The head mounted display device of  claim 1 , wherein the determined user specified operation is an application start operation, and to perform the determined user specified operation the processor is further configured to:
 select a target application program from a plurality of application programs executable by the processor based on the identified one or more semantic tags;   generate a target virtual object associated with the target application program; and   display the generated target virtual object at a virtual location based on the physical location associated with the target physical object.   
     
     
         7 . The head mounted display device of  claim 1 , wherein the processor is further configured to:
 determine a user indicated direction for a user of the head mounted display device; and   select the target virtual object or the target physical object further based on the determined user indicated direction.   
     
     
         8 . The head mounted display device of  claim 7 , wherein the user indicated direction is determined based on a detected gaze direction of the user or a detected hand gesture of the user. 
     
     
         9 . The head mounted display device of  claim 1 , further comprising a deep neural network processor configured to:
 receive scene data of the physical environment captured by the camera device;   detect a physical object in the physical environment;   recognize the physical object based on a trained artificial intelligence machine learning model; and   store a reference to the recognized physical object in the database with an associated semantic tag.   
     
     
         10 . The head mounted display device of  claim 9 , wherein the processor is further configured to:
 receive a user input directed to the detected physical object in the physical environment, the user input including a user specified semantic tag; and   associate the detected physical object with the user specified semantic tag in the database.   
     
     
         11 . The head mounted display device of  claim 1 , wherein the processor is further configured to:
 determine a confidence value for the selection of the target virtual object or the target physical object;   based on determining that the confidence value is below a threshold value, present a query to the user for a user confirmation of the selection of the target virtual object or the target physical object.   
     
     
         12 . A method comprising:
 at a processor:
 storing a database of physical objects in a physical environment and virtual objects that have been associated with one or more semantic tags; 
 receiving a natural language input from a user via an input device; 
 performing semantic processing on the natural language input to determine a user specified operation and identify one or more semantic tags indicated by the natural language input; 
 selecting a target virtual object and a target physical object from the physical objects and virtual objects in the database based on the identified one or more semantic tags; 
 performing the determined user specified operation on the target virtual object based on the target physical object; and 
 displaying the target virtual object at a physical location associated with the target physical object. 
   
     
     
         13 . The method of  claim 12 , wherein the physical objects are selected from the group consisting of a room, a wall, a room furnishing, a table, a chair, a surface, physical location, and a person. 
     
     
         14 . The method of  claim 12 , wherein the physical location of the target physical object is outside of a field of view of a camera device of a head mounted display device that includes the processor. 
     
     
         15 . The method of  claim 12 , wherein the natural language input is a voice input received via an input device. 
     
     
         16 . The method of  claim 12 , wherein the determined user specified operation is a move operation, and performing the determined user specified operation further comprises:
 updating a location of the target virtual object in the physical environment based on the physical location associated with the target physical object; and   displaying the target virtual object at the updated location via the display device.   
     
     
         17 . The method of  claim 12 , wherein the determined user specified operation is an application start operation, and performing the determined user specified operation further comprises:
 selecting a target application program from a plurality of application programs executable by the processor based on the identified one or more semantic tags;   generating a target virtual object associated with the target application program; and   displaying the generated target virtual object at a virtual location based on the physical location associated with the target physical object.   
     
     
         18 . The method of  claim 12 , further comprising:
 receiving scene data of the physical environment captured by a camera device;   detecting a physical object in the physical environment;   recognizing the physical object based on a trained artificial intelligence machine learning model; and   storing a reference to the recognized physical object in the database with an associated semantic tag.   
     
     
         19 . The method of  claim 18 , further comprising:
 receiving a user input directed to the detected physical object in the physical environment, the user input including a user specified semantic tag; and   associating the detected physical object with the user specified semantic tag in the database.   
     
     
         20 . A computer device comprising:
 a display device configured to display virtual objects at locations in a physical environment;   a camera device configured to capture images of the physical environment;   an input device configured to receive a user input; and   a processor configured to:
 receive a natural language input from a user via the input device; 
 perform semantic processing on the natural language input to determine a user specified operation and identify one or more semantic tags indicated by the natural language input; 
 select a target virtual object and a target physical object based on the identified one or more semantic tags; 
 perform the determined user specified operation on the target virtual object based on the target physical object; and 
 display the target virtual object via the display device.