Hybrid augmented reality multimodal operation neural integration environment

Abstract

A method of controlling a device relative to one or more objects in an environment of a user employing the device may include receiving a volitional input from the user indicative of a task to be performed relative to an object with the device, receiving object targeting information associated with interaction between the device and the object where the object targeting information is presented in an augmented reality context, integrating the volitional input with the object targeting information to determine a control command to direct the device to interact with the object, and providing the control command to the device.

What is claimed is: 
     
         1 . A method of controlling a device relative to an environment, the method comprising:
 receiving a volitional input from a user indicative of a task to be performed relative to an object with the device;   receiving object targeting information associated with performing the task by the device, wherein the object targeting information comprises eye tracking information of the user that is indicative of a position of the object in the environment;   generating a control command based on a combination of the volitional input and the object targeting information, wherein the control command is configured to direct the device to interact with the object; and   providing the control command to the device.   
     
     
         2 . The method of  claim 1 , wherein the receiving the volitional input comprises receiving brain-machine interface command inputs from the user. 
     
     
         3 . The method of  claim 1 , wherein the object targeting information further comprises at least one of a size, shape, or orientation of the object, or information indicative of a color, texture, or inertia of the object. 
     
     
         4 . The method of  claim 1 , wherein to interact with the object is based on the task, the position of the object, and at least one of a size, shape, or orientation of the object. 
     
     
         5 . The method of  claim 1 , wherein receiving the object targeting information further comprises receiving real-time feedback and updating the control command based on the real-time feedback. 
     
     
         6 . The method of  claim 1 , wherein the eye tracking information is based on an alignment of a measured gaze location of the user's eye with the object. 
     
     
         7 . The method of  claim 1 , wherein the receiving the object targeting information comprises receiving the eye tracking information via goggles or glasses worn by the user. 
     
     
         8 . The method of  claim 1 , wherein the environment is a virtual or augmented reality environment and the object is a virtual object in the virtual or augmented reality environment. 
     
     
         9 . The method of  claim 8 , wherein the virtual object is a plurality of menu options, and the virtual object is displayed in the virtual or augmented reality environment. 
     
     
         10 . The method of  claim 9 , wherein to interact with the virtual object is to select one of the plurality of menu options. 
     
     
         11 . The method of  claim 1 , wherein the environment is an environment of the user and the object is a physical object in the environment of the user. 
     
     
         12 . The method of  claim 1 , wherein the device is a computer, laptop, or mobile computing device. 
     
     
         13 . The method of  claim 1 , further comprising:
 receiving a machine vision input for the object;   identifying the object targeting information of the object from the machine vision input; and   recognizing the object based on the identified object targeting information.   
     
     
         14 . The method of  claim 13 , wherein the generating the control command is further based on the recognized object. 
     
     
         15 . A computer system comprising:
 a controller, wherein the controller is configured to:
 receive a volitional input from a user indicative of a task to be performed relative to an object in an environment; 
 receive object targeting information associated with performing the task by the computer system, wherein the object targeting information comprises eye tracking information of the user that is indicative of a position of the object in the environment; 
 generate a control command based on a combination of the volitional input and the object targeting information, wherein the control command is configured to direct the computer system to interact with the object; and 
 provide the control command to the computer system. 
   
     
     
         16 . The computer system of  claim 15 , wherein the controller is further configured to receive brain-machine interface command inputs from the user. 
     
     
         17 . The computer system of  claim 15 , wherein the object targeting information further comprises at least one of a size, shape, or orientation of the object, or information indicative of a color, texture, or inertia of the object. 
     
     
         18 . The computer system of  claim 15 , wherein to interact with the object is based on the task, the position of the object, and at least one of a size, shape, or orientation of the object. 
     
     
         19 . The computer system of  claim 15 , wherein the controller is further configured to receive real-time feedback and to update the control command based on the real-time feedback. 
     
     
         20 . The computer system of  claim 15 , wherein the eye tracking information is based on an alignment of a measured gaze location of the user's eye with the object. 
     
     
         21 . The computer system of  claim 15 , wherein the controller is further configured to receive the eye tracking information via goggles or glasses worn by the user. 
     
     
         22 . The computer system of  claim 15 , wherein
 the object is a virtual object and the environment is a virtual or augmented reality environment, and   the computer system further comprises a display configured to display the virtual object in the virtual or augmented reality environment.   
     
     
         23 . The computer system of  claim 22 , wherein the virtual object is a plurality of menu options. 
     
     
         24 . The computer system of  claim 23 , wherein to interact with the virtual object is to select one of the plurality of menu options. 
     
     
         25 . The computer system of  claim 15 , wherein
 the object is a physical object detected by machine vision,   the environment is an environment of the user, and   the computer system further comprises a machine vision unit configured to detect the physical object in the environment of the user.   
     
     
         26 . The computer system of  claim 25 , wherein the controller is further configured to:
 receive a machine vision input for the object from the machine vision unit;   identify the object targeting information of the object from the machine vision input; and   recognize the object based on the identified object targeting information.   
     
     
         27 . The computer system of  claim 26 , wherein the control command is further based on the recognized object.