Touch Free Interface for Augmented Reality Systems

Abstract

A method and system for augmented reality. Images of a real world scene are obtained from one or more from one or more image sensors. An orientation and/or location of the image sensors is obtained from one or more state sensors. A real world object at which a predefined pointing object is performing a predefined gesture in the images of the real world scene is identified and data associated with the identified object is displayed on a viewing device. The invention also provides a computer program comprising computer program code means for performing all the steps of the method of the invention when said program is run on a computer.

1 - 56 . (canceled) 
     
     
         57 . An augmented reality device, comprising:
 at least one processor configured to:   receive, from an image sensor, image information associated with a scene;   output, to a display, visual data to enable a user to view both the visual data and at least a portion of the scene; and   detect, in the image information, a predefined gesture performed by the user;   correlate the predefined gesture with the visual data; and   alter the displayed visual data based on the correlation.   
     
     
         58 . The augmented reality device of  claim 57 , wherein the visual data includes at least one of information associated with one or more objects in the scene, images associated with the scene, and one or more distances associated with the scene. 
     
     
         59 . The augmented reality device of  claim 57 , wherein, to correlate the predefined gesture with the visual data, the at least one processor is configured to:
 determine a reference location in three dimensional space of at leas portion of a hand of the user; and   determine in at least one of the visual data and the image information, data associated with the reference location.   
     
     
         60 . The augmented reality device of  claim 59 , wherein, to alter the displayed visual data, the at least one processor is configured to change the visual data as a function of the data associated with the reference location. 
     
     
         61 . The augmented reality device of  claim 57 , wherein, the predefined gesture is a hand gesture. 
     
     
         62 . The augmented reality device of  claim 61 , wherein the hand gesture includes a pointing finger. 
     
     
         63 . The augmented reality device of  claim 57 , wherein the at least one processor is further configured to output at least one of a message and a command based on the correlation. 
     
     
         64 . The augmented reality device of  claim 57 , wherein the predefined gesture is performed by a predefined object, and further wherein the at least one processor is further configured to:
 calibrate one or more parameters associated with the predefined object; and   detect the predefined gesture using the one or more parameters.   
     
     
         65 . The augmented reality device of  claim 57 , wherein the at least one processor is further configured to enable a user to choose one or more filters that screen what data is selected as the visual data. 
     
     
         66 . The augmented reality device of  claim 57 , wherein the at least one processor is further configured to:
 detect at least one object pointed to by the user; and   determine the visual data based on the detected object.   
     
     
         67 . The augmented reality device of  claim 57 , wherein the display is a head-mounted display. 
     
     
         68 . The augmented reality device of  claim 67 , wherein the at least one processor is further configured to:
 determine an orientation of the head-mounted display;   detect, using the determined orientation, at least one object pointed to by the user; and   determine the visual data based on the detected object.   
     
     
         69 . The augmented reality device of  claim 67 , wherein the at least one processor is further configured to:
 determine an orientation of the head-mounted display;   determine at least one fingertip location in the image information;   detect, using the determined orientation and the at least one fingertip location, at least one object pointed to by the user; and   determine the visual data based on the detected object.   
     
     
         70 . The augmented reality device of  claim 57 , wherein the predefined gesture is performed by a predefined object, and further wherein, to output the visual data to the display, the at least one processor is further configured to prevent data from being displayed at one or more locations associated with the predefined object. 
     
     
         71 . The augmented reality device of  claim 57 , wherein the at least one processor is further configured to:
 stop display of the visual data based on a head motion of the user; and   resume display of the visual data when the head motion is below a predefined motion speed.   
     
     
         72 . The augmented reality device of  claim 57 , wherein the visual data provides an indication that information related to an object in the scene is available. 
     
     
         73 . The augmented reality device of  claim 57 , wherein to output the visual data to the display, the at least one processor is further configured to superimpose the visual data on at least a portion of the user's view of the scene. 
     
     
         74 . A non-transitory computer-readable medium comprising instructions that, when executed by at least one processor, cause the at least one processor to perform operations including:
 receiving, from an image sensor, image information associated with a scene;   outputting, to a display, visual data to enable a user to view both the visual data and at least a portion of the scene; and   detecting, in the image information, a predefined gesture performed by the user;   correlating the predefined gesture with the visual data; and   altering the displayed visual data based on the correlation.   
     
     
         75 . The non-transitory computer-readable medium of  claim 74 , wherein the visual data includes at least one of information associated with one or more objects in the scene, images associated with the scene, and one or more distances associated with the scene. 
     
     
         76 . The non-transitory compute readable medium of  claim 74 , wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform operations including:
 determining a reference location in three dimensional space of at least a portion of a hand of the user; and   determining, in at least one of the visual data and the image information, data associated with the reference location.   
     
     
         77 . The non-transitory computer-readable medium of  claim 74 , wherein the predefined gesture is performed by a predefined object, and further wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform operations including:
 calibrating one or more parameters associated with the predefined object; and   detecting the predefined gesture using the one or more parameters.   
     
     
         78 . The non-transitory computer-readable medium of  claim 74 , wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform an operation including enabling a user to choose one or more filters that screen what data is selected as the visual data. 
     
     
         79 . The non-transitory computer-readable medium of  claim 74 , wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform operations including:
 detecting at least one object pointed to by the user; and   determining the visual data based on the detected object.   
     
     
         80 . The non-transitory computer-readable medium of  claim 74 , wherein the display is a head-mounted display. 
     
     
         81 . The non-transitory computer-readable medium of  claim 80 , wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform operations including:
 determining an orientation of the head-mounted display;   detecting, using the determined orientation, at least one object pointed to by the user; and   determining the visual data based on the detected object.   
     
     
         82 . The non-transitory computer-readable medium of  claim 80 , wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform operations including:
 determining an orientation of the head-mounted display;   determining at least one fingertip location in the image information;   detecting, using the determined orientation and the at least one fingertip location, at least one object pointed to by the user; and   determining the visual data based on the detected object.   
     
     
         83 . The non-transitory computer-readable medium of  claim 74 , wherein the predefined gesture is performed by a predefined object, and further wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform an operation including preventing data from being displayed at one or more locations associated with the predefined object. 
     
     
         84 . The non-transitory computer-readable medium of  claim 74 , wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform operations including:
 stopping display of the visual data based on a head motion of the user; and   resuming display of the visual data when the head motion is below a predefined motion speed.   
     
     
         85 . The non-transitory computer-readable medium of  claim 74 , wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform an operation including superimposing the visual data on at least a portion of the user's view of the scene. 
     
     
         86 . A augmented reality method, comprising:
 receiving, from an image sensor, image information associated with a scene;   outputting, to a display, visual data to enable a user to view both the visual data and at least a portion of the scene; and   detecting, in the image information, a predefined gesture performed by the user;   correlating the predefined gesture with the visual data; and   altering the displayed visual data based on the correlation.