Systems and methods for biomechanically-based eye signals for interacting with real and virtual objects

Abstract

Systems and methods are provided for discerning the intent of a device wearer primarily based on movements of the eyes. The system can be included within unobtrusive headwear that performs eye tracking and controls screen display. The system can also utilize remote eye tracking camera(s), remote displays and/or other ancillary inputs. Screen layout is optimized to facilitate the formation and reliable detection of rapid eye signals. The detection of eye signals is based on tracking physiological movements of the eye that are under voluntary control by the device wearer. The detection of eye signals results in actions that are compatible with wearable computing and a wide range of display devices.

We claim: 
     
         1 . A method for providing a graphical user interface to determine intent of a user based at least in part on movement of the user's eye or eyes using a detector, comprising:
 identifying, with the detector, when at least one eye of the user is directed at a starting location;   identifying, with the detector, a first saccade of the at least one eye from the starting location towards a first object;   confirming, with the detector, that the first saccade of the at least one eye is completed to within a predetermined distance from the location of the first object;   presenting one or more choice objects on a display at locations different than the location of the first object; and   thereafter monitoring, with the detector, the at least one eye to identify whether the at least one eye performs a second saccade towards one of the one or more choice object locations.   
     
     
         2 . The method of  claim 1 , further comprising:
 confirming, with the detector, that the second saccade of the at least one eye is completed to within a predetermined distance from one of the one or more choice objects thereby identifying a selected choice object; and   performing an action related to one or more of the starting location, an object at the starting location, the first object, and the selected choice object.   
     
     
         3 . The method of  claim 2 , wherein the action is performed on all or a portion of an object at the starting location. 
     
     
         4 . The method of  claim 1 , wherein an object at the starting location and the first object are one of physical objects and virtual objects presented on the display. 
     
     
         5 . The method of  claim 1 , wherein the first object is designed to avoid attracting attention of the user. 
     
     
         6 . The method of  claim 5 , wherein, to avoid attracting attention of the user, the first object is one or more of small relative to other objects on the display, displayed with low contrast relative to other objects on the display, displayed with low contrast relative to a background on the display, displayed with low luminance relative to other objects on the display, displayed with low transparency relative to other objects on the display, positioned away from a central region of the display, familiar to the user, and repeatedly positioned at a same location on the display. 
     
     
         7 . The method of  claim 1 , wherein the first object is removed from the user's view before, upon, or following completion of the first saccade to within a predetermined distance from the location of first object. 
     
     
         8 . The method of  claim 7 , wherein removing the first object from the view of the user produces a gap effect. 
     
     
         9 . The method of  claim 1 , wherein the second saccade is a memory-guided saccade. 
     
     
         10 . The method of  claim 1 , wherein identifying a second saccade further comprises determining a velocity of the eye movement and, based on the speed and direction of eye movement, predicting that a destination for the eye movement is within a predetermined distance from a location of the choice object. 
     
     
         11 . The method of  claim 10 , wherein the action is performed prior to completion of the second saccade. 
     
     
         12 . The method of  claim 2 , wherein the action comprises removing choice objects from the display. 
     
     
         13 . The method of  claim 2 , wherein the action comprises presenting one or more additional choice objects on a display at locations different than the selected choice object location. 
     
     
         14 . The method of  claim 13 , further comprising:
 confirming, with the detector, that a third saccade of the at least one eye is completed to within a predetermined distance from the location of one of the one or more additional choice objects and within a predetermined time, thereby identifying an additional selected choice object; and   performing an additional action in relation to one or more of the starting location, an object at the starting location, the first object, the selected choice object, and the additional selected choice object.   
     
     
         15 . The method of  claim 14 , wherein the third saccade is a memory-guided saccade. 
     
     
         16 . The method of  claim 1 , wherein thereafter monitoring the at least one eye comprises:
 monitoring the at least one eye to confirm that the at least one eye is not directed within a predetermined distance at any of the one or more choice object locations within a predetermined time; and   removing the one or more choice objects from the display.   
     
     
         17 . The method of  claim 2 , wherein the selected choice object is a “home” icon, and wherein the action comprises presenting a predetermined screen on the display. 
     
     
         18 . The method of  claim 2 , wherein the selected choice object is a “back” icon, and wherein the action comprises undoing the effects of a previous action. 
     
     
         19 . The method of  claim 1 , whereby the one or more choice objects are familiar to the user, thereby reducing cognitive load. 
     
     
         20 . A system for providing a graphical user interface to determine intent of a user based at least in part on movement of the user's eye or eyes, comprising:
 a detector configured to monitor movement of at least one eye of the user;   an electronic display operatively associated with the detector;   a processing unit operatively coupled to the detector and electronic display; and   a module stored in memory to:
 identify, with the detector, when at least one eye is directed at a starting location; 
 identify, with the detector, a first saccade of the at least one eye from the starting location towards a first object; 
 confirm, with the detector, that the first saccade of the at least one eye is completed to within a predetermined distance from the location of the first object; 
 present one or more choice objects on a display at locations different than the location of the first object; and 
 thereafter monitor, with the detector, the at least one eye to identify whether the at least one eye performs a second saccade towards one of the one or more choice object locations. 
   
     
     
         21 . The system of  claim 20 , wherein the module is further configured to:
 confirm, with the detector, that the second saccade of the at least one eye is completed to within a predetermined distance from the location of one of the one or more choice objects, thereby identifying a selected choice object; and   perform an action related to one or more of the starting location, an object at the starting location, the first object, and the selected choice object.   
     
     
         22 . The system of  claim 20 , wherein the module is further configured to:
 monitor the at least one eye to confirm that the at least one eye is not directed within a predetermined distance at any of the one or more choice object locations within a predetermined time; and   remove the one or more choice objects from the display.   
     
     
         23 . The system of  claim 20 , wherein the display is one of a head-mounted display, virtual reality display, augmented reality display, mixed reality display and remote display. 
     
     
         24 . The system of  claim 20 , further comprising a mobile electronic device, wherein the display is provided on the mobile electronic device, and wherein the detector comprises a camera on the display oriented such that user viewing the display is within a field-of-view of the camera. 
     
     
         25 . A method for providing a graphical user interface to determine intent of a user based at least in part on movement of the user's eye or eyes using a detector, comprising:
 identifying, with the detector, when at least one eye of the user is directed at a starting location;   identifying, with the detector, a first saccade of the at least one eye from the starting location towards a first object;   confirming, with the detector, that the first saccade of the at least one eye is completed to within a predetermined distance from the location of the first object;   identifying, with the detector, a second saccade of the at least one eye from the first object towards a second object;   confirming, with the detector, that the second saccade of the at least one eye is completed to within a predetermined distance from the location of the second object; and   performing an action related to one or more of the starting location, an object at the starting location, the first object, and the second object.