Eye tracking based on vergence

Abstract

An eye tracking technique obtains a more precise estimate of a gaze location on an image by determining the vergence location of a user's eyes. The more precise estimate of the gaze location may be obtained by using multiple inputs, including a coarse estimate of the gaze location using a camera that receives non-visible light reflected from the eyes, discrete probabilities of gaze locations for each eye, depth information for objects contained in the image, and saliency information for objects contained in the image.

What is claimed is: 
     
       1. A computer-implemented method for a head mounted device, the computer-implemented method comprising:
 obtaining right-eye position data corresponding to a right eye; 
 obtaining left-eye position data corresponding to a left-eye; 
 generating a two-dimensional (2D) joint-eye probability map from the right-eye position data and the left-eye position data, wherein the 2D joint-eye probability map includes a plurality of gaze location probabilities corresponding to locations in a field-of-view (FOV); and 
 determining a vergence location of the right and left eyes from the plurality of locations of the 2D joint-eye probability map, wherein the vergence location is further determined based on at least one characteristic of at least one object in an image presented by the head mounted device. 
 
     
     
       2. The computer-implemented method of  claim 1 , wherein determining the vergence location includes selecting one or more of the locations, corresponding to highest gaze location probabilities among the gaze location probabilities, in the 2D joint-eye probability map as the vergence location. 
     
     
       3. The computer-implemented method of  claim 1 , wherein the at least one characteristic of the at least one object in the image comprises respective depths of a plurality of objects in the image, and wherein the computer-implemented method further comprises:
 analyzing a depth map corresponding to the plurality of objects in the image, wherein the plurality of objects include a first object and a second object, and wherein the analysis of the depth map determines that the first and second objects are at different depths in the image; 
 determining that one of the left eye or the right eye is gazing at the first object; and 
 in response to determination that the one of the left eye or the right eye is gazing at the first object, reducing gaze location probabilities for locations in the 2D joint-eye probability map that represent the other of the left eye or the right eye gazing at the second object. 
 
     
     
       4. The computer-implemented method of  claim 3 , wherein the locations in the 2D joint-eye probability map, which have their gaze location probabilities reduced, correspond to locations within edges of the second object. 
     
     
       5. The computer-implemented method of  claim 1 , wherein the at least one characteristic of the at least one object in the image comprises saliency of interest points in the image, and wherein the computer-implemented method further comprises:
 analyzing a saliency map corresponding to the image, wherein the analysis of the saliency map identifies saliency locations for high-frequency interest points in the image; 
 increasing gaze location probabilities for locations in the 2D joint-eye probability map corresponding to the saliency locations; and 
 decreasing gaze locations probabilities for locations in the 2D joint-eye probability map corresponding to low-frequency interest points in the image. 
 
     
     
       6. The computer-implemented method of  claim 1 , wherein the right-eye position data includes images of the right eye, wherein the left-eye position data includes images of the left eye, and wherein the images of the right and left eyes are generated by a camera of the head mounted device that receives non-visible light reflected from the right and left eyes. 
     
     
       7. The computer-implemented method of  claim 1 , further comprising in response to determination of the vergence location:
 performing at least one action related to operation of the head mounted device, wherein the at least one action includes one or more of:
 applying foveated rendering to the image, wherein the foveated rendering is applied at the vergence location; 
 operating a gaze-contingent user interface (UI) at the vergence location; 
 performing gaze-contingent tone mapping; and 
 performing gaze-contingent image quality corrections for the image. 
 
 
     
     
       8. The computer-implemented method of  claim 1 , wherein generating the joint-eye probability map from the right-eye data and the left-eye data includes:
 generating a first 2D discrete-eye probability map from the left-eye data; and 
 generating a second 2D discrete-eye probability map from the right-eye data, wherein the first and second 2D discrete-eye probability maps include a plurality of discrete gaze location probabilities for the left eye and the right eye, respectively, and 
 wherein the joint-eye probability map is generated by combining the discrete gaze location probabilities from the first and second 2D discrete-eye probability maps. 
 
     
     
       9. A head mounted device, comprising:
 an eye tracking system configured to generate right-eye position data corresponding to a right eye and left-eye position data corresponding to a left eye; 
 processing logic coupled to the eye tracking system; and 
 one or more memories coupled to the processing logic, wherein the one or more memories store instructions that in response to execution by the processing logic, cause the head mounted device to perform operations to:
 generate a two-dimensional (2D) joint-eye probability map from the right-eye position data and the left-eye position data, wherein the 2D joint-eye probability map includes a plurality of gaze location probabilities corresponding to locations in a field-of-view (FOV); and 
 determine a vergence location of the right and left eyes from the plurality of locations of the 2D joint-eye probability map, wherein the vergence location is further determined based on at least one characteristic of at least one object in the FOV. 
 
 
     
     
       10. The head mounted device of  claim 9 , wherein the operations to determine the vergence location includes operations to select one or more of the locations, corresponding to highest gaze location probabilities among the gaze location probabilities, in the 2D joint-eye probability map as the vergence location. 
     
     
       11. The head mounted device of  claim 9 , wherein the at least one characteristic of the at least one object in the FOV comprises respective depths of a plurality of objects in the FOV, and wherein the one or more memories store instructions that in response to execution by the processing logic, further cause the head mounted device to perform operations to:
 analyze a depth map of the plurality of objects in the FOV, wherein the objects include a first object and a second object, and wherein the analysis of the depth map determines that the first and second objects are at different depths in the FOV; 
 determine that one of the left eye or the right eye is gazing at the first object; and 
 in response to determination that the one of the left eye or the right eye is gazing at the first object, reduce gaze location probabilities for locations in the 2D joint-eye probability map that represent the other of the left eye or the right eye gazing at the second object. 
 
     
     
       12. The head mounted device of  claim 9 , wherein the at least one characteristic of the at least one object in the FOV comprises saliency of interest points in the FOV, and wherein the one or more memories store instructions that in response to execution by the processing logic, further cause the head mounted device to perform operations to:
 analyze a saliency map corresponding to the FOV, wherein the analysis of the saliency map identifies saliency locations for high-frequency interest points in the image FOV; 
 increase gaze location probabilities for locations in the 2D joint-eye probability map corresponding to the saliency locations; and 
 decrease gaze locations probabilities for locations in the 2D joint-eye probability map corresponding to low-frequency interest points in the FOV. 
 
     
     
       13. The head mounted device of  claim 9 , wherein the operations to generate the joint-eye probability map from the right-eye data and the left-eye data include operations to:
 generate a first 2D discrete-eye probability map from the left-eye data; and 
 generate a second 2D discrete-eye probability map from the right-eye data, wherein the first and second 2D discrete-eye probability maps include a plurality of discrete gaze location probabilities for the left eye and the right eye, respectively, and 
 wherein the joint-eye probability map is generated by combining the discrete gaze location probabilities from the first and second 2D discrete-eye probability maps. 
 
     
     
       14. The head mounted device of  claim 9 , further comprising a display configured to present an image having the determined vergence location of the right and left eyes. 
     
     
       15. A head mounted device, comprising:
 a near-eye optical system configured to present an image to left and right eyes of a user; 
 an eye tracking system configured to generate right-eye position data corresponding to the right eye and left-eye position data corresponding to the left eye; and 
 a processing system, coupled to the near-eye optical system and to the eye tracking system, configured to:
 perform a first eye tracking evaluation to obtain an estimate of a gaze location of the left and right eyes on the image presented by the near-eye optical system, wherein the first eye tracking evaluation obtains the estimate of the gaze location by using the right-eye position data and the left-eye position data generated by the eye tracking system; and 
 perform a second eye tracking evaluation that refines the estimate of the gaze location, wherein the second eye tracking evaluation uses a vergence location of the left and right eyes to obtain the refined estimate of the gaze location, and wherein the vergence location is disambiguated based on at least one of depth information or saliency information, associated with a plurality of objects in the image presented by the near-eye optical system. 
 
 
     
     
       16. The head mounted device of  claim 15 , wherein to perform the second eye tracking evaluation, the processing system is configured to:
 generate a two-dimensional (2D) joint-eye probability map from the right-eye position data and the left-eye position data, wherein the 2D joint-eye probability map includes a plurality of gaze location probabilities corresponding to locations in a field-of-view (FOV) that includes the image presented by the near-eye optical system; and 
 determine the vergence location of the right and left eyes from the plurality of locations of the 2D joint-eye probability map. 
 
     
     
       17. The head mounted device of  claim 16 , wherein to perform the second eye tracking evaluation, the processing system is further configured to:
 analyze a depth map corresponding to the depth information associated with the plurality of objects in the image presented by the near-eye optical system, wherein the plurality of objects include a first object and a second object, and wherein the analysis of the depth map determines that the first and second objects are at different depths in the image; 
 determine that one of the left eye or the right eye is gazing at the first object; and 
 in response to determination that the one of the left eye or the right eye is gazing at the first object, reduce gaze location probabilities for locations in the 2D joint-eye probability map that represent the other of the left eye or the right eye gazing at the second object, 
 wherein the locations in the 2D joint-eye probability map, which have their gaze location probabilities reduced, correspond to locations within edges of the second object. 
 
     
     
       18. The head mounted device of  claim 16 , wherein to perform the second eye tracking evaluation, the processing system is further configured to:
 analyze a saliency map corresponding to the saliency information associated with the plurality of objects in the image being presented by the near eye optical system, wherein the analysis of the saliency map identifies saliency locations for high-frequency interest points in the image; 
 increase gaze location probabilities for locations in the 2D joint-eye probability map corresponding to the saliency locations; and 
 decrease gaze locations probabilities for locations in the 2D joint-eye probability map corresponding to low-frequency interest points in the image. 
 
     
     
       19. The head mounted device of  claim 16 , wherein to generate the joint-eye probability map from the right-eye data and the left-eye data, the processing system is configured to:
 generate a first 2D discrete-eye probability map from the left-eye data; and 
 generate a second 2D discrete-eye probability map from the right-eye data, wherein the first and second 2D discrete-eye probability maps include a plurality of discrete gaze location probabilities for the left eye and the right eye, respectively, and 
 wherein the joint-eye probability map is generated by the processing system by combining the discrete gaze location probabilities from the first and second 2D discrete-eye probability maps. 
 
     
     
       20. The head mounted device of  claim 15 , wherein the near-eye optical system is further configured to present a scene outside of the head mounted device to the left and right eyes of the user, and wherein the processing system is further configured to perform the first and second eye tracking evaluations with respect to objects in the scene.