Systems and methods for gaze-based media selection and editing

Abstract

Systems are presented herein, which may be implemented in a wearable device. The system is designed to allow a user to edit media images captured with the wearable device. The system employs eye tracking data to control various editing functions, whether prior to the time of capture, during the time of capture, or after the time of capture. Also presented are methods for determining which sections or regions of media images may be of greater interest to a user or viewer. The method employs eye tracking data to assign saliency to captured media. In both the system and the method, eye tracking data may be combined with data from additional sensors in order to enhance operation.

1 - 39 . (canceled) 
     
     
         40 . A system for editing media images, the system comprising:
 a wearable device;   a scene camera mounted on the device such that the scene camera captures media images of surroundings of a user;   a memory for storing the media images;   an eye tracking subsystem that projects a reference frame onto at least one eye of a user and associates the projected reference frame with a second reference frame of a display for capturing eye tracking data of the at least one eye of the user; and   one or more processors communicating with the scene camera and eye tracking subsystem for tagging media images captured by the scene camera based at least in part on the eye tracking data and for non-destructively editing the media images such that the media images are recoverable at a later time from the memory by the one or more processors.   
     
     
         41 . The system of  claim 40 , wherein the one or more processors are configured to assign a validity weight to the eye tracking data based on a graceful degradation in eye tracking precision. 
     
     
         42 . The system of  claim 40 , further comprising a transceiver or other wireless communication interface communicating with the one or more processors to transmit the media images to a remote location. 
     
     
         43 . The system of  claim 40 , further comprising a transceiver or other wireless communication interface communicating with the one or more processors to receive data from one or more sensors remote from the wearable device. 
     
     
         44 . A method for selecting or editing media images from a wearable device worn by a user, comprising:
 capturing media images of the user's surroundings at a scene camera of the wearable device;   capturing eye tracking data of at least one eye of the user at a user-facing camera of the wearable device;   projecting a reference frame onto the at least one eye of the user and associating the projected reference frame with a second reference frame of a display at an eye tracking subsystem of the wearable device;   at least one of selecting and editing the media images based at least in part on actions of the at least one eye identified from the eye tracking data;   detecting a threshold event based on an accelerometer of the wearable device; and   wherein at least one of selecting and editing the media images comprises automatically tagging the media images for a predetermined time immediately before the threshold event.   
     
     
         45 . The method of  claim 44 , further comprising determining a saliency value of the media images from the scene camera based at least in part on eye movement of the user identified in the eye tracking data. 
     
     
         46 . The method of  claim 45 , wherein detecting the threshold event selectively overrides other saliency determinations. 
     
     
         47 . The method of  claim 44 , further comprising recognizing predetermined faces within the media images. 
     
     
         48 . The method of  claim 44 , further comprising:
 receiving, at a communication interface of the wearable device, metrics or heuristics from a remote location;   wherein at least one of selecting and editing the media images comprises assigning saliency to one or more segments of the media images based upon the metrics or heuristics.   
     
     
         49 . The method of  claim 45 , further comprising using data from social media applications as input for partial determination of saliency in the media images. 
     
     
         50 . The method of  claim 44 , further comprising cropping media images to boundaries suggested by a user's hand motions to create a picture frame. 
     
     
         51 . The method of  claim 44 , further comprising assigning saliency to media images based at least in part on input from multiple sensors, such as squinting, smiling or laughter. 
     
     
         52 . The method of  claim 44 , further comprising automatically cropping media images to a region in which the at least one eye of the user is fixating. 
     
     
         53 . The method of  claim 44 , further comprising automatically cropping media images to a region bounded by saccadic movements of the at least one eye of the user. 
     
     
         54 . A method, comprising:
 projecting a point of light onto a user's eye;   capturing eye tracking data comprising a glint from a reflection of the point of light from the user's eye at an eye tracking camera of a wearable device;   determining movement of the user's eye relative to the glint;   capturing media images of the user's surroundings at a scene camera of the wearable device; and   editing the media images based on movement of the user's eye relative to the glint.   
     
     
         55 . The method of  claim 54 , further comprising detecting a threshold event based on an accelerometer of the wearable device. 
     
     
         56 . The method of  claim 55 , further comprising determining a saliency value of the media images from the scene camera based at least in part on eye movement of the user identified in the eye tracking data. 
     
     
         57 . The method of  claim 56 , wherein detecting the threshold event selectively overrides other saliency determinations. 
     
     
         58 . The method of  claim 54 , further comprising recognizing predetermined faces within the media images. 
     
     
         59 . The method of  claim 54 , further comprising:
 receiving, at a communication interface of the wearable device, metrics or heuristics from a remote location;   wherein editing the media images comprises assigning saliency to one or more segments of the media images based upon the metrics or heuristics.