Automated preview generation for video entertainment content

Abstract

A respective set of features, including emotion-related features, are extracted from segments of a video for which a preview is to be generated. A subset of the segments is chosen using the features and filtering criteria including at least one emotion-based filtering criterion. Respective weighted preview-suitability scores are assigned to the segments of the subset using at least a metric of similarity between individual segments and a plot summary of the video. The scores are used to select and combine segments to form a preview for the video.

What is claimed is: 
     
       1. A system, comprising:
 one or more computing devices; 
 wherein the one or more computing devices include instructions that upon execution on or across the one or more computing devices:
 obtain, via one or more programmatic interfaces from an entertainment content provider, an indication of (a) a video for which a preview is to be generated automatically, (b) audio associated with the video, and (c) a text plot summary of the video; 
 divide the video into a plurality of segments; 
 extract, from individual segments of the plurality of segments using one or more machine learning models, a respective set of features, including at least a first emotion-related feature associated with analysis of visual content of a segment, a second emotion-related feature associated with analysis of audio of the segment, and a first metric of similarity between the segment and the text plot summary; 
 select a subset of segments of the plurality of segments using at least some features of the respective sets of features and a collection of filtering criteria, wherein the collection of filtering criteria includes at least one emotion-based filtering criterion; 
 assign, to individual segments of the subset, respective weighted preview-suitability scores using a combination of metrics, wherein the combination of metrics includes the first metric of similarity between the individual segments and the text plot summary; 
 combine at least some segments of the subset, identified from the subset using at least the respective weighted preview-suitability scores, to generate a proposed preview for the video; and 
 cause the proposed preview to be presented via an interactive editing interface. 
 
 
     
     
       2. The system as recited in  claim 1 , wherein the one or more machine learning models include a multi-lingual emotion recognition model trained using input in a plurality of languages. 
     
     
       3. The system as recited in  claim 1 , wherein the one or more machine learning models including a model which generates, within a particular embedding space, (a) a first representation of audio or video of a segment of the plurality of segments and (b) a second representation of the plot summary, wherein the first metric of similarity is generated using the first representation and the second representation. 
     
     
       4. The system as recited in  claim 1 , wherein the one or more computing devices include further instructions that upon execution on or across the one or more computing devices:
 obtain a script, subtitles, or an audio content transcription for at least a particular segment of the video, and wherein a set of features extracted for the particular segment includes at least one feature derived from the script, subtitles, or the audio content transcription. 
 
     
     
       5. The system as recited in  claim 1 , wherein the one or more computing devices include further instructions that upon execution on or across the one or more computing devices:
 obtain, via the interactive editing interface, input indicating that a particular segment included in the proposed preview is to be shortened, extended or eliminated; and 
 store a modified version of the proposed preview in accordance with the input. 
 
     
     
       6. A computer-implemented method, comprising:
 extracting, from individual ones of a plurality of segments of a video for which a preview is to be generated automatically, a respective set of features, including at least a first emotion-related feature associated with analysis of visual content of a segment and a second emotion-related feature associated with analysis of audio content of the segment; 
 selecting a subset of segments of the plurality of segments using at least some features of the respective sets of features and a collection of filtering criteria, wherein the collection of filtering criteria includes at least one emotion-based filtering criterion; 
 assigning, to individual segments of the subset, respective weighted preview-suitability scores using a combination of metrics, wherein the combination of metrics includes a first metric of similarity between the individual segments and a plot summary of the video; and 
 combining, using at least the respective weighted preview-suitability scores, at least some segments of the subset to generate a first proposed preview for the video. 
 
     
     
       7. The computer-implemented method as recited in  claim 6 , further comprising:
 dividing the video into the plurality of segments using an algorithm that determines segment boundaries based at least in part on analysis of differences in color properties among successive frames of the video. 
 
     
     
       8. The computer-implemented method as recited in  claim 6 , further comprising:
 dividing the video into a sequence of chronological portions, wherein individual ones of the chronological portions include one or more segments of the plurality of segments, and wherein the combining comprises including, in the first proposed preview, at least one segment from a selected subset of chronological portions. 
 
     
     
       9. The computer-implemented method as recited in  claim 8 , wherein in accordance with a spoiler avoidance policy, the selected subset of chronological portions does not include at least one chronological portion of the sequence. 
     
     
       10. The computer-implemented method as recited in  claim 6 , wherein extracting the respective set of features comprises utilizing one or more machine learning models including a multi-lingual emotion recognition model trained using input in a plurality of languages. 
     
     
       11. The computer-implemented method as recited in  claim 6 , wherein extracting the respective set of features comprises utilizing one or more machine learning models including a model which generates, within a particular embedding space, (a) a first representation of audio or video of a segment of the plurality of segments and (b) a second representation of the plot summary, wherein the first metric of similarity is generated using the first representation and the second representation. 
     
     
       12. The computer-implemented method as recited in  claim 6 , wherein the collection of filtering criteria comprises a first filtering criterion for segments comprising faces and a second filtering criterion for segments that do not include faces, the computer-implemented method further comprising:
 determining, using one or more machine learning models, that (a) a particular segment of the plurality of segments includes a face and (b) another segment of the plurality of segments does not include a face, wherein selecting the subset of segments comprises:
 applying the first filtering criterion to the particular segment and 
 applying the second filtering criterion to the other segment. 
 
 
     
     
       13. The computer-implemented method as recited in  claim 6 , further comprising:
 obtaining an indication, via a programmatic interface, of a genre of the video; and 
 utilizing the genre to exclude at least some segments of the plurality of segments from the first proposed preview. 
 
     
     
       14. The computer-implemented method as recited in  claim 6 , further comprising:
 determining a duration of the first proposed preview based at least in part on a first preview consumption target device type; and 
 generating, corresponding to a second preview consumption target device type, a second proposed preview of the video, wherein a duration of the second proposed preview differs from a duration of the first proposed preview. 
 
     
     
       15. The computer-implemented method as recited in  claim 6 , further comprising:
 determining a first profile of a first targeted video consumer audience, wherein at least one segment of the plurality of segments is included in the first proposed preview based on the first profile; and 
 generating a second proposed preview of the video, based at least in part on a second profile of a second targeted video consumer audience. 
 
     
     
       16. One or more non-transitory computer-accessible storage media storing program instructions that when executed on or across one or more processors:
 extract, from individual ones of a plurality of segments of a video for which a preview is to be generated automatically, a respective set of features, including at least a first emotion-related feature associated with analysis of visual content of a segment and a second emotion-related feature associated with analysis of audio content of the segment; 
 select a subset of segments of the plurality of segments using at least some features of the respective sets of features and a collection of filtering criteria, wherein the collection of filtering criteria includes at least one emotion-based filtering criterion; 
 assign, to individual segments of the subset, respective weighted preview-suitability scores using a combination of metrics, wherein the combination of metrics includes a first metric of similarity between the individual segments and a plot summary of the video; and 
 combine, using at least the respective weighted preview-suitability scores, at least some segments of the subset to generate a first proposed preview for the video. 
 
     
     
       17. The one or more non-transitory computer-accessible storage media as recited in  claim 16 , wherein extraction of the respective set of features comprises utilizing one or more machine learning models including a multi-lingual emotion recognition model trained using input in a plurality of languages. 
     
     
       18. The one or more non-transitory computer-accessible storage media as recited in  claim 16 , wherein extraction of the respective set of features comprises utilizing one or more machine learning models including a model which generates a description of a particular segment. 
     
     
       19. The one or more non-transitory computer-accessible storage media as recited in  claim 16 , wherein the collection of filtering criteria comprises a first filtering criterion for segments comprising faces and a second filtering criterion for segments that do not include faces, and wherein the one or more non-transitory computer-accessible storage media store further program instructions that when executed on or across the one or more processors:
 determine, using one or more machine learning models, that (a) a particular segment of the plurality of segments includes a face and (b) another segment of the plurality of segments does not include a face, wherein selection of the subset of segments comprises:
 applying the first filtering criterion to the particular segment and 
 applying the second filtering criterion to the other segment. 
 
 
     
     
       20. The one or more non-transitory computer-accessible storage media as recited in  claim 16 , storing further program instructions that when executed on or across the one or more processors:
 obtain an indication via a programmatic interface, of a genre of the video; and 
 utilize the genre to exclude at least some segments of the plurality of segments from the first proposed preview.