System for Annotating Media Content for Automatic Content Understanding

Abstract

A system for annotating frames in a media stream frames includes a pattern recognition system (PRS) to generate PRS output metadata for a frame; an archive for storing ground truth metadata (GTM); a device to merge the GTM and PRS output metadata and thereby generate proposed annotation data (PAD); and a user interface for use by the HA. The user interface includes an editor and an input device used by the HA to approve GTM for the frame. An optimization system receives the approved GTM and metadata output by the PRS, and adjusts input parameters for the PRS to minimize a distance metric corresponding to a difference between the GTM and PRS output metadata.

We claim: 
     
         1 . A system to annotate media content, comprising:
 a pattern recognition system (PRS) having an initial set of input parameters that generates PRS output metadata associated with a frame of a media stream;   an archive for storing ground truth metadata (GTM) associated with the same frame of the media stream;   a device to merge the GTM and the PRS output metadata and thereby generate proposed annotation data (PAD); and   a user interface for use by a human annotator (HA) including
 an editor and 
 an input device to approve or edit the PAD for the frame; and 
   an optimization system to adjust input parameters for the PRS to minimize a distance metric corresponding to a difference between the GTM and PRS output metadata.   
     
     
         2 . The system of  claim 1  wherein the GTM is obtained from one or more of third party metadata, archived media stream and the HA. 
     
     
         3 . The system of  claim 2  wherein a time delay between third party metadata and the media stream is corrected by alignment. 
     
     
         4 . The system of  claim 2  including a communication network to enable a plurality of HA's to interface with the same media stream. 
     
     
         5 . The system of  claim 2  wherein when the PAD is approved it is converted to GTM. 
     
     
         6 . The system of  claim 5  wherein when the PAD is approved, it is graphically overlayed on the media stream. 
     
     
         7 . The system of  claim 1  wherein the optimization system adjusts the PRS initial set of input parameters to minimize the difference between the GTM and PRS output metadata thereby increasing accuracy. 
     
     
         8 . The system of  claim 1  wherein the PRS includes a set of state variables stored as a temporal group adjustable as a group in response to GTM. 
     
     
         9 . A method comprising:
 receiving data from a media stream, the data organized into frames;   processing the data using a pattern recognition system (PRS);   storing a state of the PRS;   generating metadata associated with the frame using the PRS;   receiving input characterized as ground truth metadata (GTM), into an optimization system;   adjusting input parameters for the PRS to minimize a distance metric corresponding to a difference between the GTM and PRS output metadata.   
     
     
         10 . The method of  claim 9  wherein said input is obtained from one or more of archived media streams, third party metadata and one or more human annotators. 
     
     
         11 . The method of  claim 10  wherein subsequent to receiving said input, said GTM and said metadata associated with said PRS are temporally aligned. 
     
     
         12 . The method of  claim 10  wherein said GTM and said metadata associated with said PRS are continuously stored and memory and periodically stored to disk thereby enabling fast recovery from system failure. 
     
     
         13 . A method comprising
 receiving from a human annotator (HA), via a human annotator user interface (HAUI), information regarding a time point selected by the HA on a timeline of the media stream;   merging existing ground truth metadata (GTM) relating to a media frame corresponding to the selected time point with pattern recognition system (PRS) output metadata relating to said media frame, thereby generating proposed annotation data (PAD) for the media frame;   displaying the media frame and the PAD to the HA;   receiving input from the HA including correction and/or approval of the PAD, where approved PAD is characterized as new GTM related to the selected time point;   storing the new GTM;   comparing the PRS output metadata and the new GTM related to the selected time point; and   adjusting PRS input parameters so that a distance metric corresponding to a difference between the new GTM and PRS output metadata related to the selected time point is minimized.   
     
     
         14 . The method of  claim 13  wherein said GTM is obtained from one or more of archived media streams, third party metadata, said human annotators and other human annotators. 
     
     
         15 . The method of  claim 14  wherein when said human annotator approves said PAD, said PAD is graphically overlaid on said media stream. 
     
     
         16 . A method comprising:
 generating output metadata associated with a frame of a media stream, output by a pattern recognition system (PRS);   storing in an archive input from a human annotator (HA) related to the frame, characterized as ground truth metadata (GTM);   merging the GTM and the PRS output metadata to thereby generate proposed annotation data (PAD); and   displaying the PAD to the HA by a user interface;   receiving via the user interface an input from the HA indicating approval of the GTM for the frame; and   adjusting input parameters for the PRS using an optimization system, to minimize a distance metric corresponding to a difference between the GTM and the PRS output metadata.   
     
     
         17 . The method of  claim 16  wherein said GTM is obtained from one or more of archived media streams, third party metadata, said human annotators and other human annotators. 
     
     
         18 . The method of  claim 17  wherein when said human annotator approves said PAD, said PAD is graphically overlaid on said media stream.