TV Content Segmentation, Categorization and Identification and Time-Aligned Applications

Abstract

A content segmentation, categorization and identification method on consumer devices (clients) is described. Methods for content tracking are illustrated that are suitable for large scale deployment and applications such as broadcast monitoring, novel content publishing and interaction. Time-aligned (synchronous) applications such as multi-language selection, customized advertisements, second screen services and content monitoring applications can be economically deployed at large scales. The client performs fingerprinting, scene change detection, audio turn detection, and logo detection on incoming video and gathers database search results, logos and text to identify and segment video streams into content, promos, and commercials. A learning engine is configured to learn rules for optimal identification and segmentation at each client for each channel and program. Content sensed at the client site is tracked with reduced computation and applications are executed with timing precision. A method and user interface for time-aligned publishing of content and subsequent usage and interaction on one or more displays is described.

1 .- 20 . (canceled) 
     
     
         21 . A method to provide content publishing services that are personalized for different user devices, the method comprising:
 detecting a logo in an incoming video;   identifying the incoming video based on the identified logo;   collecting a first set of information regarding the identified incoming video that is personalized for use by a first user device;   collecting a second set of information regarding the identified incoming video that is personalized for use by a second user device; and   adjusting content publishing services for the first user device and for the second user device according to the personalized information received by each user device.   
     
     
         22 . The method of  claim 21  further comprising:
 generating video signatures on initial frames of the incoming video; and 
 searching for the generated video signatures in opening sequences of programs stored in a reference database to further identify the incoming video. 
 
     
     
         23 . The method of  claim 21 , wherein the incoming video is a live broadcast. 
     
     
         24 . The method of  claim 21  further comprising:
 generating first time alignment information by calculating locations in position and in time for pixel offsets along an x-axis and along a y-axis between signatures generated for incoming video frames received on the first user device and corresponding matching signatures generated for an opening sequence of frames from a program found in a reference database, wherein the first time alignment information is used on the first user device to synchronize publishing content selected from the reference database with the incoming video frames. 
 
     
     
         25 . The method of  claim 21  further comprising:
 generating second time alignment information by calculating locations in position and in time for pixel offsets along an x-axis and along a y-axis between signatures generated for incoming video frames received on the second user device and corresponding matching signatures generated for an opening sequence of frames from a program found in a reference database, wherein the second time alignment information is used on the second user device to synchronize publishing content selected from the reference database with the incoming video frames. 
 
     
     
         26 . The method of  claim 21 , wherein the content publishing services for the first user device comprise a first language customization. 
     
     
         27 . The method of  claim 26 , wherein first time alignment information is generated in the first user device to synchronize the first language customization with the incoming video. 
     
     
         28 . The method of  claim 21 , wherein the content publishing services for the second user device further comprise a second language customization. 
     
     
         29 . The method of  claim 28 , wherein second time alignment information is generated in the second user device to synchronize the second language customization with the incoming video. 
     
     
         30 . A method to provide time aligned language presentations personalized for user devices, the method comprising:
 identifying audio content and video content based on a logo detected in broadcast content of a broadcast program;   determining an audio time alignment of the audio content and a video time alignment of the video content on a first user device relative to the broadcast content;   synchronizing the audio content with the video content according to the audio time alignment and the video time alignment on the first user device;   substituting the synchronized audio content with a first selected language on the first user device; and   overlaying text and the logo with customized text and a customized logo in the first selected language in the synchronized video content on the first user device.   
     
     
         31 . The method of  claim 30  further comprising:
 determining an audio time alignment of the audio content and a video time alignment of the video content on a second user device relative to the broadcast content; 
 synchronizing the audio content with the video content according to the audio time alignment and the video time alignment on the second user device; 
 substituting the synchronized audio content with a second selected language on the second user device; and 
 overlaying text and the logo with customized text and a customized logo in the second selected language in the synchronized video content on the second user device. 
 
     
     
         32 . The method of  claim 30  further comprising:
 generating audio and video signatures on initial frames of the incoming video; and 
 searching for the generated audio and video signatures in opening sequences of programs stored in a reference database to further identify the audio content and the video content. 
 
     
     
         33 . The method of  claim 30  further comprising:
 using a program guide to further identify the audio content and the video content. 
 
     
     
         34 . The method of  claim 30  further comprising:
 generating first time alignment information by calculating locations in position and in time for pixel offsets along an x-axis and along a y-axis between signatures generated for frames of the broadcast content received on the first user device and corresponding matching signatures generated for an opening sequence of frames from a program found in a reference database, wherein the first time alignment information is used on the first user device to synchronize time aligned language presentations selected from the reference database with the incoming video frames. 
 
     
     
         35 . The method of  claim 30  further comprising:
 identifying a logo change as an indicator of and a time of a scene change; 
 substituting the synchronized audio content with a selected language at the time of the scene change: and 
 overlaying text and the logo with customized text and a customized logo in the synchronized video content. 
 
     
     
         36 . The method of  claim 30  further comprising:
 monitoring content defined by the detected logo and identified audio content and video content on a plurality of user devices; and 
 sending the monitored content to a server to determine audience measurements of contents watched on the plurality of user devices. 
 
     
     
         37 . The method of  claim 36 , wherein the monitored content includes broadcast programs. 
     
     
         38 . The method of  claim 36 , wherein the monitored content includes advertisements specified as having a content logo. 
     
     
         39 . A computer readable non-transitory medium encoded with computer readable program data and code, the computer readable program data and code when executed perform a method to provide content publishing services that are personalized for different user devices, the method comprising:
 detecting a logo in an incoming video;   identifying the incoming video based on the identified logo;   collecting a first set of information regarding the identified incoming video that is personalized for use by a first user device;   collecting a second set of information regarding the identified incoming video that is personalized for use by a second user device; and   adjusting content publishing services for the first user device and for the second user device according to the personalized information received by each user device.   
     
     
         40 . The computer readable non-transitory medium method of  claim 39  further comprising:
 generating video signatures on initial frames of the incoming video; and 
 searching for the generated video signatures in opening sequences of programs stored in a reference database to further identify the incoming video.