Gaze direction mapping

Abstract

A method for determining correspondence between a gaze direction and an environment around a wearable device is disclosed. The wearable device may include an eye tracking device and an outward facing image sensor. The method may include receiving an input parameter and at least one scene image from the outward facing image sensor. The method may further include determining, with at least the eye tracking device, at least one gaze direction of a wearer of the wearable device at a point in time corresponding to when the scene image was captured by the outward facing image sensor. The method may additionally include determining, based at least in part on the input parameter, that a particular scene image includes at least a portion of a predefined image. The method may moreover include determining, based on the at least one gaze direction, at least one gaze point on the particular scene image.

1 . (canceled) 
     
     
         2 . A method for determining a correspondence between a gaze direction and an environment around a wearable device, wherein the wearable device comprises an eye tracking device, an outward facing image sensor and 3D sensor, and wherein the method comprises:
 receiving an input parameter comprising a predefined image and dynamic area data identifying at least one dynamic area within the predefined image;   building, with the 3D sensor, a 3D map of the environment around the wearable device,   receiving a scene image from the outward facing image sensor;   determining, with at least the eye tracking device, a gaze direction of a wearer of the wearable device at a point in time corresponding to when the scene image was captured by the outward facing image sensor;   determining, based at least in part on the input parameter, that the scene image includes at least a remaining portion of the predefined image wherein the remaining portion of the predefined image does not include the at least one dynamic area;   determining a position of the remaining portion in the scene image;   determining, based on the at least the gaze direction and the position, a gaze point on the predefined image, and,   mapping the gaze point onto the predefined image in the 3D map.   
     
     
         3 . The method for determining a correspondence between a gaze direction and an environment around a wearable device of  claim 2 , wherein:
 the 3D sensor comprises (i) a sensor using structured light, (ii) a time of flight sensor, and/or (iii) stereo camera imaging.   
     
     
         4 . The method for determining a correspondence between a gaze direction and an environment around a wearable device of  claim 2 , wherein:
 the wearable device further comprises a positional device which provides information relating to the orientation and position of the wearable device, and wherein the information from the 3D sensor is combined with information from the positional device.   
     
     
         5 . The method for determining a correspondence between a gaze direction and an environment around a wearable device of the  claim 4 , wherein the positional device comprises a gyroscope and/or accelerometer. 
     
     
         6 . The method for determining a correspondence between a gaze direction and an environment around a wearable device of  claim 2 , wherein the method further comprises:
 receiving dynamic area content information comprising an image; and   determining, based at least on the dynamic area content information, that the particular scene image also includes at least a portion of the image in a location of the dynamic area in the scene image.   
     
     
         7 . The method for determining a correspondence between a gaze direction and an environment around a wearable device of  claim 2 , wherein:
 the dynamic content information is correlated to the point in time corresponding to when the scene image was captured by the outward facing image sensor.   
     
     
         8 . The method for determining a correspondence between a gaze direction and an environment around a wearable device of  claim 2 , wherein the predefined image comprises: predefined text. 
     
     
         9 . The method for determining a correspondence between a gaze direction and an environment around a wearable device of  claim 2 , wherein the input parameter comprises:
 a manually mapped point in the predefined image.   
     
     
         10 . The method for determining a correspondence between a gaze direction and an environment around a wearable device of  claim 2 , wherein the input parameter comprises:
 a manually mapped area of the predefined image.   
     
     
         11 . The method for determining a correspondence between a gaze direction and an environment around a wearable device of  claim 2 , wherein the input parameter comprises:
 a manually identified presence of the predefined image in the scene image.   
     
     
         12 . The method for determining a correspondence between a gaze direction and an environment around a wearable device of  claim 2 , wherein the input parameter comprises:
 a plurality of identified instances of the predefined image in video received from the outward facing image sensor.   
     
     
         13 . A system for determining a correspondence between a gaze direction and an environment around a wearable device, wherein the system comprises:
 one or more processors configured to at least:   receive an input parameter comprising a predefined image and dynamic area data identifying at least one dynamic area within the predefined image;   building, with the 3D sensor, a 3D map of the environment around the wearable device;   receive a scene image from an outward facing image sensor of a wearable device;   determine, with at least an eye tracking device of the wearable device, a gaze direction of a wearer of the wearable device at a point in time corresponding to when the scene image was captured by the outward facing image sensor;   determine, based at least in part on the input parameter, that the scene image includes at least a remaining portion of the predefined image, wherein the remaining portion of the predefined image does not include the at least one dynamic area;   determine a position of in the scene image;   determine, based at least on the gaze direction and the position, a gaze point on the predefined image, and   mapping the gaze point onto the predefined image in the 3D map.   
     
     
         14 . The system for determining a correspondence between a gaze direction and an environment around a wearable device of  claim 13 , wherein:
 the 3D sensor comprises (i) a sensor using structured light, (ii) a time of flight sensor, and/or (iii) stereo camera imaging.   
     
     
         15 . The system for determining a correspondence between a gaze direction and an environment around a wearable device of  claim 13 , wherein:
 the wearable device further comprises a positional device which provides information relating to the orientation and position of the wearable device, and wherein the information from the 3D sensor is combined with information from the positional device.   
     
     
         16 . The method for determining a correspondence between a gaze direction and an environment around a wearable device of the previous  claim 13 , wherein the positional device comprises a gyroscope and/or accelerometer. 
     
     
         17 . The system for determining a correspondence between a gaze direction and an environment around a wearable device of  claim 13 , wherein the one or more processors are further configured to at least:
 receive dynamic area content information comprising an image; and   determine, based at least on the dynamic area content information, that the particular scene image also includes at least a portion of the image in a location of the dynamic area in the scene image.   
     
     
         18 . The system for determining a correspondence between a gaze direction and an environment around a wearable device of  claim 17 , wherein:
 the dynamic content information is correlated to the point in time corresponding to when the scene image was captured by the outward facing image sensor.   
     
     
         19 . The system for determining a correspondence between a gaze direction and an environment around a wearable device of  claim 13 , wherein the predefined image comprises:
 predefined text.   
     
     
         20 . The system for determining a correspondence between a gaze direction and an environment around a wearable device of  claim 13 , wherein the input parameter comprises:
 a manually mapped area in the predefined image.   
     
     
         21 . A non-transitory machine readable medium having instructions stored thereon for determining a correspondence between a gaze direction and an environment around a wearable device, wherein the instructions are executable by one or more processors for at least:
 receiving an input parameter comprising a predefined image and dynamic area data identifying at least one dynamic area within the predefined image;   building, with the 3D sensor, a 3D map of the environment around the wearable device;   receiving a scene image from an outward facing image sensor of a wearable device;   determining, with at least an eye tracking device of the wearable device, a gaze direction of a wearer of the wearable device at a point in time corresponding to when the scene image was captured by the outward facing image sensor;   determining, based at least in part on the input parameter, that the scene image includes at least a remaining portion of the predefined image, wherein the remaining portion of the predefined image does not include the at least one dynamic area;   determining a position of the remaining portion in the scene image;   determining, based on the at least the gaze direction and the position, a gaze point on the predefined image, and   mapping the gaze point onto the predefined image in the 3D map.