Video classification using user behavior from a network digital video recorder

Abstract

Particular embodiments provide a system to determine ad segments in a video asset to enable subsequent ad replacement in video programs. The system is included in a multiple service operator (MSO) system that broadcasts video programs via a broadcast schedule. The MSO may not know the location of the ad segments in the video asset. To determine the ad segments, the MSO uses a classifier to classify video program segments and advertisements in the video asset. The classifier may be integrated with an nDVR system. By integrating with the nDVR system, particular embodiments may determine user behavior information, such as trick play commands, from the nDVR system. The classifier may use the user behavior information to detect ad segments in the video asset. In one embodiment, the classifier may fuse outputs from different detectors to detect and validate ad segments in the video program.

1 . A method comprising:
 broadcasting a video asset using a video broadcast system based on a broadcast schedule;   recording the video asset in a network digital video recorder (nDVR) system to allow users to request the video asset using the nDVR system on demand;   performing a content analysis of first audio, video, and/or text (AVT) features of the video asset to obtain a boundary that defines a first video segment and a second video segment;   performing a content similarity analysis between second AVT features from the first video segment and third AVT features from the second video segment;   classifying the first video segment and the second video segment as either an ad segment or a video program segment based on the content analysis and the content similarity analysis; and   determining an action to be performed on the video asset based on the classifying of the first video segment and the second video segment.   
     
     
         2 . The method of  claim 1 , wherein the action comprises triggering an ad replacement system to replace the first video segment or the second video segment with a new advertisement when the first video segment or the second video segment is classified as the ad segment. 
     
     
         3 . The method of  claim 1 , wherein the action comprises triggering a data analytics system to analyze the first video segment or the second video segment to identify which advertisement is found in the first video segment or the second video segment when the first video segment or the second video segment is classified as the ad segment, wherein the data analytics system stores the user behavior information with respect to the identified advertisement. 
     
     
         4 - 5 . (canceled) 
     
     
         6 . The method of  claim 1 , wherein the content similarity analysis is used to selectively adjust a probability of the first video segment or the second video segment being the ad segment. 
     
     
         7 . (canceled) 
     
     
         8 . The method of  claim 1 , wherein the content analysis comprises analyzing the video asset for ad markers or frames that indicate ad transitions. 
     
     
         9 . The method of  claim 1  further comprising determining whether the content similarity analysis should be performed to validate the classifying of the first video segment or the second video segment as the ad segment or the video program segment. 
     
     
         10 . The method of  claim 9 , wherein the content similarity analysis is performed when a confidence score for the first video segment or the second video segment is below a threshold, the confidence score determined based on the content analysis. 
     
     
         11 . The method of  claim 1 , wherein comparing the second AVT features and the third AVT features using the content similarity analysis comprises:
 generating a first feature vector for the second AVT features;   generating a second feature vector for the third AVT features; and   comparing the first feature vector and the second feature vector using a similarity function.   
     
     
         12 . The method of  claim 1 , wherein an entity providing the video broadcast system does not receive a location of some ad segments in the video asset from a content source of the video asset. 
     
     
         13 - 19 . (canceled) 
     
     
         20 . A system comprising:
 a video broadcast system configured to broadcast a video asset based on a broadcast schedule, wherein the video asset includes video program segments and ad segments;   a network digital video recorder (nDVR) system coupled to the video broadcast system and configured to record the video asset to allow users to request the video using the nDVR system on demand;   a classifier coupled to receive user behavior information from user devices viewing the video asset using the nDVR system, the classifier configured for:
 performing a content analysis of first audio, video, and/or text (AVT) features of the video asset to obtain a boundary that defines a first video segment and a second video segment; 
 performing a content similarity analysis between second AVT features from the first video segment and third AVT features from the second video segment; 
 classifying the first video segment and the second video segment as either an ad segment or a video program segment based on the content analysis and the content similarity analysis; and 
 determining an action to be performed on the video asset based on the classifying of the first video segment and the second video segment. 
   
     
     
         21 . The method of  claim 20  wherein the action comprises triggering an ad replacement system to replace the first video segment or the second video segment with a new advertisement when the first video segment or the second video segment is classified as the ad segment. 
     
     
         22 . The method of  claim 1 , wherein the action comprises triggering a data analytics system to analyze the first video segment or the second video segment to identify which advertisement is found in the first video segment or the second video segment when the first video segment or the second video segment is classified as the ad segment, wherein the data analytics system stores the user behavior information with respect to the identified advertisement. 
     
     
         23 . The method of  claim 20  wherein the content similarity analysis is used to selectively adjust a probability of the first video segment or the second video segment being the ad segment. 
     
     
         24 . The method of  claim 20 , wherein the content analysis comprises analyzing the video asset for ad markers or frames that indicate ad transitions. 
     
     
         25 . The method of  claim 20  further comprising determining whether the content similarity analysis should be performed to validate the classifying of the first video segment or the second video segment as the ad segment or the video program segment. 
     
     
         26 . The method of  claim 20  wherein the content similarity analysis is performed when a confidence score for the first video segment or the second video segment is below a threshold, the confidence score determined based on the content analysis.