Headshot extraction and curation

Abstract

Systems and techniques for generation and curation of a professional headshot from a set of image data. The systems and techniques images from the set of image data based on characteristics of the representation of the individual within the image. The systems and techniques further include determining a bounding box to define a headshot, the bounding box determined based on guidelines established by heuristics and/or machine learning algorithms trained using data labeled based on heuristics.

The invention claimed is: 
     
       1. A method comprising:
 accessing a set of image data from a video source including a representation of one or more individuals; 
 for each individual of the one or more individuals: 
 determine an image characteristic score reflective of a quality of the representation of the individual within an image of the set of image data; 
 determine a pose of the individual using a pose estimation machine learning algorithm, the pose indicative of an estimated direction of a face of the individual related to a direction of an imaging device that captured the set of image data; 
 determining the image based on the estimated direction of the face being within a threshold amount of the direction of the imaging device and the image characteristic score being equal to or greater than a threshold amount; 
 determining, based on a bounding box location and bounding box dimensions for a headshot of the individual, a bounding box surrounding a head of the individual and the bounding box location and bounding box dimensions defined based on characteristics comprising:
 a location of the face of the individual; 
 a size of the face of the individual relative to a height of the bounding box; 
 a position of the face of the individual within the bounding box; 
 a representation of shoulders of the individual within the bounding box; 
 
 generating the headshot by placing the bounding box at the bounding box location within the image and removing image data lying outside of the bounding box; 
 determining an aesthetic score for the headshot, the aesthetic score determined based on an aggregate of scores for a set of aesthetic characteristics; 
 adjusting an aesthetic characteristic in response to the aesthetic score being below a first threshold; and 
 causing to be presented, via a user interface and in association with the video source, headshots for each individual of the one or more individuals. 
 
     
     
       2. The method of  claim 1 , wherein determining the image of the set of image data comprises identifying the pose of the individual using the pose estimation machine learning algorithm, the pose estimation machine learning algorithm configured to receive the set of image data and output the image and a score associated with the estimated direction of the face of the individual related to the direction of the imaging device, the image selected based on the score exceeding a threshold; and
 determining the bounding box for the headshot comprises using a multi-task cascaded convolutional neural network trained using headshot data labeled using heuristics describing head size, head position, and head pose of the headshot. 
 
     
     
       3. The method of  claim 1 , wherein determining the image comprises using a pose based heuristic that implements a cost function that determines a cost based on the pose and a size of a head of the individual within the bounding box. 
     
     
       4. The method of  claim 1 , wherein generating the headshot comprises:
 identifying, in response to the aesthetic score being below the first threshold, an aesthetic characteristic of the set of aesthetic characteristics to adjust, the aesthetic characteristic identified based on the aesthetic score for the aesthetic characteristic being below a second threshold, wherein the set of aesthetic characteristics comprise:
 an image tone; 
 a brightness; 
 a pose of the individual; 
 a background blur; and 
 a sharpness of the image. 
 
 
     
     
       5. A method for automatically generating professional headshots from source image data, the method comprising:
 accessing a set of image data from a video source including a representation of one or more individuals; 
 for each individual of the one or more individuals:
 determining an image of the set of image data based at least in part on a characteristic of the representation of the individual; 
 determining a bounding box location and one or more bounding box dimensions for a headshot for the individual, the bounding box surrounding a head of the individual; and 
 generating the headshot by:
 determining a frame of the image by cropping the image by placing the bounding box at the bounding box location within the image and removing image data lying outside of the bounding box; 
 
 determining an aesthetic score for the frame, the aesthetic score determined based on an aggregate of scores for a set of aesthetic characteristics; and 
 adjusting an aesthetic characteristic in response to the aesthetic score being below a first threshold; and 
 
 causing to be presented, via a user interface and in association with the video source, headshots for each individual of the one or more individuals. 
 
     
     
       6. The method of  claim 5 , wherein determining the image of the set of image data comprises localizing the individual within the image of the set of image data using a machine learning algorithm trained using labeled face detection data indicating visibility of faces within images. 
     
     
       7. The method of  claim 5 , wherein determining the bounding box comprises identifying the bounding box using a machine learning algorithm trained using headshot data labeled using heuristics describing head size, head position, and head pose of the headshot. 
     
     
       8. The method of  claim 5 , wherein generating the headshot comprises performing one or more aesthetic adjustments to the image. 
     
     
       9. The method of  claim 8 , wherein the one or more aesthetic adjustments include at least one of:
 an image tone adjustment; a brightness adjustment; 
 a pose correction; 
 a background blur adjustment; or 
 a sharpness adjustment of the image. 
 
     
     
       10. The method of  claim 5 , wherein determining the image is based at least in part on the image including the individual unobscured in a portion of the image. 
     
     
       11. The method of  claim 5 , further comprising storing the headshot in association with the set of image data; and
 accessing, by a user on a computing device viewing a video including the set of image data, the headshot by pausing the video to view the image and accessing an informational panel describing one or more individuals within the image with the headshot. 
 
     
     
       12. The method of  claim 5 , wherein determining the image comprises selecting the image based at least in part on a pose of the individual within the image. 
     
     
       13. The method of  claim 12 , wherein selecting the image based at least in part on the pose of the individual comprises determining a head pose estimation for the individual using a machine learning model, wherein the image is selected in response to the pose indicating a forward-facing head pose estimation. 
     
     
       14. The method of  claim 5 , wherein determining the bounding box is based at least in part on a set of heuristics comprising:
 a head of the individual being centered within the bounding box; 
 the head of the individual being in an upper half of the bounding box; and 
 shoulders of the individual being included within the bounding box. 
 
     
     
       15. A non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform operation comprising:
 accessing a set of image data from a video source including a representation of one or more individuals; 
 for each individual of the one or more individuals:
 determining an image of the set of image data based at least in part on a characteristic of the representation of the individual; 
 determining a bounding box location and one or more bounding box dimensions for a headshot for the individual, the bounding box surrounding a head of the individual; and 
 
 generating the headshot by:
 determining a frame of the image by placing the bounding box at the bounding box location within the image and removing image data lying outside of the bounding box; 
 
 determining an aesthetic score for the frame, the aesthetic score determined based on an aggregate of scores for a set of aesthetic characteristics; and 
 adjusting an aesthetic characteristic in response to the aesthetic score being below a first threshold; and 
 causing to be presented, via a user interface and in association with the video source, headshots for each individual of the one or more individuals. 
 
     
     
       16. The non-transitory computer-readable medium of  claim 15 , wherein determining the image comprises selecting the image based at least in part on a pose of the individual within the image. 
     
     
       17. The non-transitory computer-readable medium of  claim 16 , wherein determining the image based at least in part on the pose of the individual comprises using a pose based heuristic that implements a cost function that determines a cost based at least in part on the pose and a size of a head of the individual within the bounding box. 
     
     
       18. The non-transitory computer-readable medium of  claim 15 , wherein determining the bounding box comprises identifying the bounding box using a machine learning algorithm trained using headshot data labeled using heuristics describing head size, head position, and head pose of the individual. 
     
     
       19. The non-transitory computer-readable medium of  claim 15 , further comprising determining an aesthetic score for the headshot and performing an aesthetic adjustment in response to the aesthetic score being below a threshold. 
     
     
       20. The non-transitory computer-readable medium of  claim 15 , further comprising storing the headshot in association with the set of image data for access by a user viewing the set of image data.