Identifying, describing, and sharing salient events in images and videos

Abstract

A computing system for identifying, describing, and sharing salient events depicted in images and videos executes feature detection algorithms on multimedia input (e.g., video and/or images). The computing system applies semantic reasoning techniques to the output of the feature detection algorithms. The computing system identifies salient event segments of the multimedia input as a result of the semantic reasoning. The computing system can incorporate the salient event segments into a visual presentation, such as a video clip. Alternatively or in addition, the computing system can generate a natural language description of the content of the multimedia input.

1 . A video assistant for understanding content of a video, the video assistant embodied in one or more non-transitory machine accessible storage media of a computing system and comprising instructions executable by one or more processors to cause the computing system to:
 detect a plurality of different features in a plurality of different segments of a video by executing a plurality of different feature detection algorithms on the video, each of the video segments comprising one or more frames of the video;   determine an event evidenced by the detected features;   determine a plurality of salient activities associated with the event; and   algorithmically identify a plurality of different salient event segments of the video, each of the salient event segments depicting a salient activity associated with the event.   
     
     
         2 . The video assistant of  claim 1 , comprising instructions executable to determine salient event criteria associated with the event, and determine the salient activities associated with the event based on the salient event criteria. 
     
     
         3 . The video assistant of  claim 2 , comprising instructions executable to determine the salient event criteria by one or more of: algorithmically analyzing a video collection and algorithmically learning a user specification relating to the salient event criteria. 
     
     
         4 . The video assistant of  claim 1 , comprising instructions executable to determine a saliency indicator associated with each of the salient event segments of the video, and select a subset of the plurality of salient event segments for inclusion in a visual presentation based on the saliency indicator. 
     
     
         5 . The video assistant of  claim 1 , comprising instructions executable to extract the salient event segments from the video and incorporate the extracted salient event segments into a video clip. 
     
     
         6 . The video assistant of  claim 5 , comprising instructions executable to, in response to user input, share the video clip with another computing device over a network. 
     
     
         7 . The video assistant of  claim 5 , comprising instructions executable to one or more of: (i) by a human-computer interface device, interactively edit the video clip and (ii) automatically edit the video clip. 
     
     
         8 . The video assistant of  claim 7 , comprising instructions executable to store data relating to the interactive editing of the video clip, execute one or more machine learning algorithms on the stored data, and, in response to the execution of the one or more machine learning algorithms on the stored data, update one or more of: the determination of salient activities associated with the event and the identification of salient event segments. 
     
     
         9 . The video assistant of  claim 1 , comprising instructions executable to, by a human-computer interface device, output a natural language description of the one or more salient event segments. 
     
     
         10 . A computing system for understanding content of a video, the computing system comprising:
 one or more computing devices; and   a plurality of processor-executable modules embodied in one or more non-transitory machine accessible storage media of the one or more computing devices, the processor-executable modules comprising:   a visual content understanding module to cause the computing system to:
 detect a plurality of different features in a plurality of different segments of a video by executing one or more event recognition algorithms; 
 determine a semantic description of an event evidenced by one or more of the detected features; and 
 identify one or more salient event segments of the video, each salient event segment depicting a salient activity associated with the event; 
   an output generator module to cause the computing system to output a video clip comprising the salient event segments; and   an interactive storyboard module to cause the computing system to one or more of: interactively edit the video clip and share the video clip over a network.   
     
     
         11 . The computing system of  claim 10 , wherein the visual content understanding module is to cause the computing system to determine a configuration of a camera used to record the video; derive, from the camera configuration, a user intent with respect to the video; and identify the salient event segments by selecting one or more segments of the video that relate to the user intent. 
     
     
         12 . The computing system of  claim 11 , wherein in response to the user intent, the output generator module is to cause the computing system to select a template for creating the video clip. 
     
     
         13 . The computing system of  claim 10 , wherein the visual content understanding module is to cause the computing system to determine the semantic description based on a plurality of different algorithmically-detected features comprising two or more of: a visual feature, an audio feature, a textual feature, and a meta-level feature indicative of a camera configuration. 
     
     
         14 . The computing system of  claim 10 , wherein the visual content understanding module is to cause the computing system to determine relationships between the detected features, map the detected features and relationships to semantic concepts, and formulate the semantic description to comprise the semantic concepts. 
     
     
         15 . The computing system of  claim 10 , wherein the visual content understanding module is to cause the computing system to, in an automated fashion, associate the semantic description with the video. 
     
     
         16 . The computing system of  claim 10 , wherein the visual content understanding module is to cause the computing system to determine a salient event criterion and, based on the salient event criterion, identify the salient activity associated with the event. 
     
     
         17 . The computing system of  claim 16 , wherein the computing system is to learn the salient event criterion by analyzing a professionally-made video. 
     
     
         18 . The computing system of  claim 16 , wherein the computing system is to analyze one or more of: semantic content of a video collection and user input, and determine the salient event criterion based on the analysis of the one or more of the semantic content and the user input. 
     
     
         19 . The computing system of  claim 10 , wherein the computing system is to determine a saliency indicator comprising data associated with one or more of the detected features, and use the saliency indicator to identify the salient event segments. 
     
     
         20 . A computing system for understanding visual content in digital images, the computing system comprising:
 one or more computing devices; and   instructions embodied in one or more non-transitory machine accessible storage media of the one or more computing devices, the instructions executable by the one or more computing devices to cause the computing system to:   detect a plurality of different features in a set of digital images by executing a plurality of different feature detection algorithms on the set of images;   map the one or more features detected by the feature detection algorithms to an event, the event evidenced by the one or more detected features;   determine a plurality of salient activities associated with the event;   extract one or more salient event segments from the set of images, each of the salient event segments depicting a salient activity associated with the event; and   incorporate the extracted one or more salient event segments into a visual presentation.   
     
     
         21 . The computing system of  claim 20 , wherein the instructions cause the computing system to select at least two of: a visual feature detection algorithm, an audio feature detection algorithm, and a textual feature detection algorithm, execute the selected feature detection algorithms to detect at least two of: a visual feature, an audio feature, and a textual feature of the set of images, and determine the event evidenced by at least two of: the visual feature, the audio feature, and the textual feature. 
     
     
         22 . The computing system of  claim 20 , wherein the instructions cause the computing system to, in an automated fashion, generate a semantic description of the event based on the one or more features detected by the feature detection algorithms. 
     
     
         23 . The computing system of  claim 20 , wherein the instructions cause the computing system to determine a saliency indicator associated with each of the salient event segments, and arrange the salient event segments in the visual presentation according to the saliency indicators associated with the salient event segments. 
     
     
         24 . The computing system of  claim 23 , wherein the instructions cause the computing system to one or more of: (i) by a human-computer interface device of the computing system, interactively rearrange the salient event segments in the visual presentation and (ii) automatically rearrange the salient event segments in the visual presentation. 
     
     
         25 . The computing system of  claim 23 , wherein the instructions cause the computing system to select a subset of the salient event segments based on the saliency indicators associated with the salient event segments, and create a visual presentation comprising the salient event segments in the selected subset of salient event segments. 
     
     
         26 . The computing system of  claim 20 , wherein the instructions cause the computing system to, in an automated fashion, associate a description of the event with the images in the set of digital images. 
     
     
         27 . The computing system of  claim 26 , wherein the instructions cause the computing system to detect user input comprising a textual description, compare the textual description to the description of the event associated with the images in the set of digital images, and, in an automated fashion, suggest one or more images having a relevancy to the text description as determined by the comparison of the textual description of the user input to the description of the event.