Human-computer interface using high-speed and accurate tracking of user interactions

Abstract

Embodiments described herein relate to systems, devices, and methods for use in the implementation of a human-computer interface using high-speed, and efficient tracking of user interactions with a User Interface/User Experience that is strategically presented to the user. Embodiments described herein also relate to the implementation of a hardware agnostic human-computer interface that uses neural, oculomotor, and/or electromyography signals to mediate user manipulation of machines and devices.

1 . An apparatus, comprising:
 a display configured to present an interactive environment to a user;   an eye-tracker coupled to the display, the eye-tracker including at least two sensors, the at least two sensors being configured to record eye-movement signals from an eye of the user;   an interfacing device operatively coupled to the display and the eye-tracker, the interfacing device including:   a memory; and   a processor operatively coupled to the memory and configured to:
 receive the eye-movement signals from the at least two sensors in the eye-tracker; generate and present a stimulus, via the interactive environment and via the display, to the user; 
 determine, based on the eye-movement signals, a point of focus of the user; determine, based on the point of focus of the user, an action intended by the user; and 
 implement the action intended by the user. 
   
     
     
         2 . The apparatus of  claim 1 , wherein:
 the display includes a display lens configured to project the interactive environment to the user; and   the at least two sensors in the eye-tracker are positioned around the display lens, and along an axes.   
     
     
         3 . The apparatus of  claim 1 , wherein:
 the display includes a display lens configured to project the interactive environment to the user; and   the eye-tracker include at least four sensors, the at least four sensors being positioned around the display lens, and along two orthogonal axes.   
     
     
         4 . The apparatus of  claim 1 , wherein
 the eye-tracker is further configured to send, to the processor, eye-movement signals recorded by each sensor from the at least two sensors in an independent manner.   
     
     
         5 . The apparatus of  claim 1 , wherein:
 the eye-movement signals include a plurality of sets of eye-movement signals, each set of eye-movement signals being recorded by each sensor from the at least two sensors, each set of eye-movement signals being independent of the plurality of sets of eye-movement signals recorded by the remaining sensors from the at least two sensors, and   the processor is further configured to:   compute, based on each sets of eye-movement signals from the plurality of sets of eye-movement signals, a gaze vector associated with each sensor from the at least two sensors, the gaze vector associated with each sensor indicating a gaze angle of the eye of the user;   determine a degree of obliqueness of each gaze vector associated with each sensor from the at least two sensors, the degree of obliqueness being relative to a vertical angle associated with that sensor;   determine, based on the degree of obliqueness of each gaze vector associated with each sensor from the at least two sensors, a weight associated with each sensor from the at least two sensors, to generate a set of weights; and   apply the set of weights to the plurality of sets of eye-movement signals to determine a set of calibrated eye-movement signals.   
     
     
         6 . The apparatus of  claim 1 , wherein:
 the eye-movement signals include a plurality of sets of eye-movement signals, each set of eye-movement signals being recorded by each sensor from the at least two sensors, each set of eye-movement signals being independent of the plurality of sets of eye-movement signals recorded by the remaining sensors from the at least two sensors, and   the processor is further configured to:   compute, based on each sets of eye-movement signals from the plurality of sets of eye-movement signals, a gaze vector associated with each sensor from the at least two sensors, the gaze vector associated with each sensor indicating a gaze angle of the eye of the user;   determine a degree of obliqueness of each gaze vector associated with each sensor from the at least two sensors, the degree of obliqueness being relative to a vertical angle associated with that sensor;   determine, based on the degree of obliqueness of each gaze vector associated with each sensor from the at least two sensors and an empirically pre-determined weighting function, a weight associated with each sensor from the at least two sensors, to generate a set of weights; and   apply the set of weights to the plurality of sets of eye-movement signals to determine a set of calibrated eye-movement signals.   
     
     
         7 . The apparatus of  claim 1 , wherein:
 the eye-movement signals include a plurality of sets of eye-movement signals, each set of eye-movement signals being recorded by each sensor from the at least two sensors, each set of eye-movement signals being independent of the plurality of sets of eye-movement signals recorded by the remaining sensors from the at least two sensors, and   the processor is further configured to:
 identify a set of missing data points in the plurality of sets of eye-movement signals; 
 receive, from the eye-tracker, information related to the at least two sensors; 
   generate, based on the information related to the at least two sensors, a kinematics model of a set of simulated eye-movements of a simulated user;   compute, based on the kinematics model, a plurality of sets of simulated eye-movement signals, each set of simulated eye-movement signals being associated with each sensor from the at least two sensors;   compute a set of replacement data points to replace the set of missing data points in the eye-movement signals received from the at least two sensors, based on the plurality of sets of simulated eye-movement signals; and   incorporate the set of replacement data points to replace the set of missing data points and to generate calibrated eye-movement signals associated with each sensor from the at least two sensors, the point of focus of the user being determined based on the calibrated eye-movement signals.   
     
     
         8 . The apparatus of  claim 1 , wherein the eye-tracker includes at least four sensors, the four sensors being positioned along two orthogonal axes, and the processor is further configured to compute, based on the eye-movements signals, a set of gaze vectors subtended relative to the two orthogonal axes, the set of gaze vectors configured to collectively represent a gaze angle of the eye of the user. 
     
     
         9 . The apparatus of  claim 1 , wherein the eye-movement signals correspond to a gaze angle of the eye of the user at a first time point, and the processor is further configured to determine, based on the eye-movement signals, a gaze angle of the eye of the user at a second time point different from the first time point, the second time point occurring after the first time point. 
     
     
         10 . The apparatus of  claim 1 , wherein the eye-movement signals correspond to a gaze angle of the eye of the user at a first time point, and are associated with a first measure of momentum at the first time point, and the processor is further configured to determine, based on the eye-movement signals, the gaze angle at the first time point, and the first measure of momentum at the first time point, a gaze angle of the eye of the user at a second time point different from the first time point, the second time point occurring after the first time point. 
     
     
         11 . The apparatus of  claim 1 , further comprising a neural recording device configured to record neural signals generated by the user, the neural signals including electroencephalogram (EEG) signals, wherein the point of focus is a calculated point of focus and the processor is further configured to:
 receive the EEG signals, the EEG signals including at least one of visually evoked potentials (VEP), auditory evoked potentials (AEP), motor imagery signals, Event Related Potentials (ERP), and brain state dependent signals;   determine, based on the EEG signals, an expected point of focus of the user;   compute, based on a comparison between the calculated point of focus and the expected point of focus, a measure of error associated with the calculated point of focus;   correct the calculated point of focus, based on the measure of error, to generate a calibrated point of focus of the user.   
     
     
         12 . A non-transitory processor-readable medium storing code representing instructions to be executed by a processor, the instructions comprising code to cause the processor to:
 generate an interactive user environment that can be manipulated, by a user, to perform a set of actions;   define a set of stimuli that can be presented to the user via the interactive user environment;   present, via a display, at least one stimulus from the set of stimuli to the user;   receive, from an eye-tracker, eye-movement signals generated by the user;   automatically calibrate the eye-movement signals based on information related to the presented stimulus, to generate a set of calibrated eye-movement signals;   determine, based on the set of calibrated eye-movement signals and the stimulus presented, a point of focus of the user;   determine, based on the point of focus, an action intended by the user; and   implement the action via the interactive user environment.   
     
     
         13 . The non-transitory processor-readable medium of  claim 12 , wherein the code to automatically calibrate the eye-movement signals includes code to cause the processor to:
 present a grid of objects in three dimensional space;   present a graphical indicator at a first location, and configured to direct the point of focus of the user to the first location;   determine, based on the presentation of the graphical indicator, an expected point of focus of the user;   determine, based on the eye-movement signals, an actual point of focus of the user; and   compute, based on a comparison of the expected point of focus and the actual point of focus, a measure of reliability of the eye-tracker, the measure of reliability being used to generate the set of calibrated eye-movement signals.   
     
     
         14 . The non-transitory processor-readable medium of  claim 13 , wherein the code to automatically calibrate the eye-movement signals includes code to cause the processor to:
 generate the grid of objects in three dimensional space, the grid being configured to have a predetermined density of objects, the predetermined density corresponding to an indication of granularity of the measure of reliability of the eye-tracker; and   generate a density control configured to allow modification of a value of the predetermined density of objects.   
     
     
         15 . The non-transitory processor-readable medium of  claim 12 , wherein the code to automatically calibrate the eye-movement signals includes code to cause the processor to:
 generate a spatial map of reliability of the eye-tracker, the spatial map configured to correspond to a spatial region of the display.   
     
     
         16 . The non-transitory processor-readable medium of  claim 12 , wherein the code to automatically calibrate the eye-movement signals includes code to cause the processor to:
 define a set of covert fixation stimuli configured to have a high likelihood of being the focus of attention of the user, the set of covert fixation stimuli having higher visual salience;   present at least one covert fixation stimulus to the user at a first location;   determine, based on the presentation of the covert fixation stimulus, an expected point of focus of the user;   determine, based on the eye-movement signals, an actual point of focus of the user; and   compute, based on a comparison of the expected point of focus and the actual point of focus, the set of calibrated eye-movement signals.   
     
     
         17 . The non-transitory processor-readable medium of  claim 12 , wherein the code to automatically calibrate the eye-movement signals includes code to cause the processor to:
 define a set of covert fixation stimuli configured to have a high likelihood of being the focus of attention of the user, the set of covert fixation stimuli having at least one of increased contrast in luminance, dynamic movement, or increased spatial frequency;   present at least one covert fixation stimulus to the user at a first location;   determine, based on the presentation of the covert fixation stimulus, an expected point of focus of the user;   determine, based on the eye-movement signals, an actual point of focus of the user; and   compute, based on a comparison of the expected point of focus and the actual point of focus, the set of calibrated eye-movement signals.   
     
     
         18 . The non-transitory processor-readable medium of  claim 12 , the code to automatically calibrate the eye-movement signals includes code to cause the processor to:
 present a scaling-bias calibration stimulus configured to prompt a visual search by the user;   receive, from the eye-tracker, a set of calibration eye-movement signals generated by the visual search by the user;   determine, based on the set of calibration eye-movement signals, a first maximum and a first minimum gaze position of the user along a first axis;   determine, based on the set of calibration eye-movement signals, a second maximum and a second minimum gaze position of the user along a second axis orthogonal to the first axis; and   compute, based on the first maximum and first minimum gaze positions and the second maximum and the second minimum gaze positions, and the scaling-bias calibration stimulus, a measure of scaling and a measure of bias associated with a set of eye-movements of the user, the set of calibrated eye-movement signals being generated based on the measure of scaling and the measure of bias.   
     
     
         19 . The non-transitory processor-readable medium of  claim 18 , wherein at least one of the measure of scaling and a measure of bias are configured to have an exponential relationship with a measure of eccentricity associated with gaze position along the first axis or the second axis. 
     
     
         20 . The non-transitory processor-readable medium of  claim 12 , wherein the code to automatically calibrate the eye-movement signals includes code to cause the processor to:
 present a three-dimensional stimulus including at least one interactive object at a first location relative to an eye of the user;   extract a set of smooth-pursuit signals from the eye-movement signals, the smooth-pursuit signals indicating a trajectory of the point of focus of the eye of the user corresponding to the first location relative to the body of the user;   receive, from a body-tracker, a trajectory of body movement of the user;   determine, based on the eye-movement signals, a calculated trajectory of the point of focus of the user;   determine, based on the first location relative to the user and the trajectory of body movement, an expected trajectory of the point of focus of the user; and   determine, based on a comparison between the calculated trajectory and the expected trajectory, a measure of accuracy associated with the determination of point of focus of the user, the set of calibrated eye-movement signals being generated based on the measure of measure of accuracy associated with the determination of point of focus of the user.   
     
     
         21 - 30 . (canceled)