Systems and techniques for estimating eye pose

Abstract

An eye tracking system can include eye-tracking camera(s) configured to obtain image(s) of the eye at different exposure times or different frame rates. For example, image(s) of the eye taken with a longer exposure time can be analyzed to detect eye feature(s) such as iris or pupil features, and image(s) taken with a shorter exposure time can be analyzed to detect glints reflected from the eye. The shorter exposure images may be taken at a higher frame rate than the longer exposure images for more accurate gaze prediction based on glint analysis, e.g., to provide glint locations to subpixel accuracy. Such glint analysis may also take into account estimated location(s) of partially or totally occluded glint(s). The longer exposure images can be analyzed for pupil center, eye center of rotation, or other characteristics. The system can predict gaze direction, e.g., for foveated rendering by a wearable display system.

What is claimed is: 
     
         1 . A head-mounted system comprising:
 one or more cameras configured to capture images of an eye of a user;   one or more light sources configured to illuminate the eye of the user to produce glints in the images of the eye that are captured by the one or more cameras; and   one or more processors operatively coupled to the one or more cameras and the one or more light sources, the one or more processors configured to:
 receive an image of the eye from the one or more cameras; 
 determine a set of one or more detected glint locations including a glint location for each of one or more glints detected in the image; 
 in response to determining that a number of locations included in the set of one or more detected glint locations is less than a minimum value:
 based at least partly on the set of one or more detected glint locations, determine a set of one or more estimated glint locations for the image, wherein the set of one or more estimated glint locations includes a glint location for each of one or more glints that is undetected in the image and predicted to be present; and 
 determine a pose of the eye of the user based at least in part on the set of one or more detected glint locations and the set of one or more estimated glint locations. 
 
   
     
     
         2 . The system of  claim 1 , wherein the one or more processors are further configured to:
 in response to determining that the number of locations included in the set of one or more detected glint locations is at least the minimum value, determine the pose of the eye of the user based at least in part on the set of one or more detected glint locations.   
     
     
         3 . The system of  claim 1 , wherein the minimum value is based at least partly on a number of the one or more light sources. 
     
     
         4 . The system of  claim 1 , wherein the minimum value is the number of the one or more light sources. 
     
     
         5 . The system of  claim 1 , wherein determining the set of one or more estimated glint locations includes applying the set of one or more detected glint locations to a model. 
     
     
         6 . The system of  claim 1 , wherein the one or more processors are further configured to determine a location of a center of a pupil of the eye in the image, and wherein determining the set of one or more estimated glint locations is further based on the determined location of the center of the pupil. 
     
     
         7 . The system of  claim 6 , wherein determining the set of one or more estimated glint locations further comprises:
 based at least partly on the determined location of the center of the pupil, determine a plurality of regions in the image of the eye in which at least one is expected to be present.   
     
     
         8 . The system of  claim 7 , wherein a number of the regions is at least the minimum value. 
     
     
         9 . The system of  claim 1 , wherein the one or more light sources are one or more infrared light sources. 
     
     
         10 . A method performed by one or more processors of a head-mounted system, the method comprising:
 receiving an image of an eye of a user, the image captured by one or more cameras of the head-mounted system;   determining a set of one or more detected glint locations including a glint location for each of one or more glints detected in the image;   in response to determining that a number of locations included in the set of one or more detected glint locations is less than a minimum value:
 based at least partly on the set of one or more detected glint locations, determining a set of one or more estimated glint locations for the image, wherein the set of one or more estimated glint locations includes a glint location for each of one or more glints that is undetected in the image and predicted to be present; and 
 determining a pose of the eye of the user based at least in part on the set of one or more detected glint locations and the set of one or more estimated glint locations. 
   
     
     
         11 . The method of  claim 10 , further comprising:
 in response to determining that the number of locations included in the set of one or more detected glint locations is at least the minimum value, determining the pose of the eye of the user based at least in part on the set of one or more detected glint locations.   
     
     
         12 . The method of  claim 10 , wherein the minimum value is based at least partly on a number of the one or more light sources. 
     
     
         13 . The method of  claim 10 , wherein the minimum value is the number of the one or more light sources. 
     
     
         14 . The method of  claim 10 , wherein determining the set of one or more estimated glint locations includes applying the set of one or more detected glint locations to a model. 
     
     
         15 . The method of  claim 10 , further comprising:
 determining a location of a center of a pupil of the eye in the image, wherein determining the set of one or more estimated glint locations is further based on the determined location of the center of the pupil.   
     
     
         16 . The method of  claim 15 , wherein determining the set of one or more estimated glint locations further comprises:
 based at least partly on the determined location of the center of the pupil, determining a plurality of regions in the image of the eye in which at least one is expected to be present.   
     
     
         17 . The method of  claim 16 , wherein a number of the regions is at least the minimum value. 
     
     
         18 . One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more processors of a head-mounted system, instruct the one or more processors to perform operations comprising:
 receiving an image of an eye of a user, the image captured by one or more cameras of the head-mounted system;   determining a set of one or more detected glint locations including a glint location for each of one or more glints detected in the image;   in response to determining that a number of locations included in the set of one or more detected glint locations is less than a minimum value:
 based at least partly on the set of one or more detected glint locations, determining a set of one or more estimated glint locations for the image, wherein the set of one or more estimated glint locations includes a glint location for each of one or more glints that is undetected in the image and predicted to be present; and 
 determining a pose of the eye of the user based at least in part on the set of one or more detected glint locations and the set of one or more estimated glint locations.