System and method for seamless content insertion on network content using audio-video fingerprinting and watermarking

Abstract

System and method is proposed to extract audio-video fingerprint and embed an audio-video watermark onto a content asset that is transmitted as part of network content. It is proposed to identify uniquely the specific content to be replaced and the point of local content insertion by looking for the embedded audio-video watermark and match the audio-video fingerprint. The identified point of insertion enables seamless local content insertion.

1 . A method for seamless content insertion of one or more local content assets, over one or more network/stored playout content assets, wherein a point of insertion is identified using an in-band marker constructed by a combination of an audio-video watermark and a unique fingerprint, used to customize content transmitted over one or more varied delivery networks, comprising the steps of:
 a. Embedding a unique network content asset identifier “AMAGI-ID” as an audio-video watermark, into the network content asset, the watermarked network content asset being labeled “AMAGI-WMED-CONTENT”;   b. Extracting a unique audio-video fingerprint “AMAGI-FP” from the network content asset;   c. Storing “AMAGI-ID” and “AMAGI-FP” in a data store “AMAGI-ID-DB”;   d. Transmitting “AMAGI-WMED-CONTENT” over one or more networks in one or more content formats;   e. Introducing a delay in the delivery of the network in order to identify the watermark and the fingerprint during network reception;   f. Monitoring the network/stored playout content assets at the time of reception to detect “AMAGI-WMED-CONTENT”;   g. Upon detecting previously inserted watermarks (in step a) in “AMAGI-WMED-CONTENT”:
 i. Extracting the corresponding “AMAGI-ID” from the network content asset; 
 ii. Extracting corresponding fingerprints for the network content asset; and 
 iii. Matching the extracted fingerprints obtained in step g(ii) with one or more previously stored “AMAGI-FP” in the “AMAGI-ID-DB”; 
   h. Upon finding a match in step g(iii):
 b) Delivering the local content asset at the correct position in the stream of network/stored playout content assets, thereby replacing the network content asset with a suitable local content asset. 
   
     
     
         2 . A method for seamless content insertion of one or more local content assets, over one or more network/stored playout content assets using an in-band marker of  claim 1  wherein:
 a. One or more watermarking methods using spread-spectrum, amplitude modulation or using quantization of signals, can be used; 
 b. The watermark uses the network content asset as the carrier channel; 
 c. The watermark effects minimal distortion in the audio and video within the network content asset; and 
 d. The watermark is independent of the delivery networks, which might have different compression and/or decompression of the audio-video content, and yet the watermark is reliably transmitted. 
 
     
     
         3 . A method for seamless content insertion of one or more local content assets, over one or more network/stored playout content assets using an in-band marker of  claim 1  wherein:
 a. One or more fingerprinting methods using perceptual characteristics of audio or video streams within one or more network/stored playout content assets, can be used to extract a unique bit pattern that at best uniquely identifies the network/stored playout content assets; and 
 b. Fingerprinting methods used utilize a hash function or a low complexity compression of audio and or video for the purpose of unique identification of the content, rather than near-exact reproduction; 
 
     
     
         4 . A method for seamless content insertion of one or more local content assets, over one or more network/stored playout content assets using an in-band marker of  claim 1  wherein if the fingerprint extracted in step g(ii) match with a unique “AMAGI-FP” in “AMAGI-ID-DB”, it is used along with the previously inserted watermark detected to identify the position of insertion of local content assets in the stream of network/stored playout content assets and the steps of watermarking and fingerprinting can be performed interchangably. 
     
     
         5 . A method for seamless content insertion of one or more local content assets, over one or more network/stored playout content assets using an in-band marker of  claim 1  wherein the delay introduced in step 1(e) is determined as the maximum value between:
 a. The duration elapsing during extraction of the audio-video watermark; and 
 b. The duration elapsing during matching the extracted fingerprint. 
 
     
     
         6 . A method for seamless content insertion of one or more local content assets, over one or more network/stored playout content assets using an in-band marker of  claim 1  wherein:
 a. An “AMAGI-BARCODE-IN” module performs steps a-d; and 
 b. An “AMAGI-BARCODE-OUT” module performs steps f and g. 
 
     
     
         7 . A method for seamless content insertion of one or more local content assets, over one or more network/stored playout content assets using an in-band marker of  claim 1  wherein the watermark is extracted from both audio and video of the network content asset or from either one of audio or video to identify the existence of “AMAGI-ID” watermark. 
     
     
         8 . A method for seamless content insertion of one or more local content assets, over one or more network/stored playout content assets using an in-band marker of  claim 1  wherein the fingerprint matching is exercised across −5 to +10 video frames, from the position of identification of the watermark in order to obtain the exact position of the point of insertion. 
     
     
         9 . A method for seamless content insertion of one or more local content assets, over one or more network/stored playout content assets using an in-band marker of  claim 1  wherein one or more fingerprint matching procedures to identify different audio and video insertion points. 
     
     
         10 . A method for seamless content insertion of one or more local content assets, over one or more network/stored playout content assets using an in-band marker of  claim 1  wherein either one of audio or video fingerprint matching and use the point of insertion identified for either audio or video to estimate the point of insertion for both audio and video. 
     
     
         11 . A method for seamless content insertion of one or more local content assets, over one or more network/stored playout content assets using an in-band marker of  claim 1  used in cable head-ends, set-top boxes, internet browsers and internet content delivery networks using audio-video fingerprinting and water marking with (a) “AMAGI-BARCODE-IN” module, (b) an entertainment content play out server, (c) an ad content play out server, (d) traffic-billing system, (e) one or more encoder or decoder, (f) “AMAGI-BARCODE-OUT” module and (g) cable and satellite TV plants configured to support a plurality of networks, in a plurality of content formats, further comprising the steps of:
 a. Performing in the “AMAGI-BARCODE-IN” module the steps of:
 i. Embedding a unique network content asset identifier “AMAGI-ID” as an audio-video watermark, into the network content asset, the watermarked network content asset being labeled “AMAGI-WMED-CONTENT”; 
 ii. Extracting a unique audio-video fingerprint “AMAGI-FP” from the network content asset; 
 iii. Storing “AMAGI-ID” and “AMAGI-FP” in a data store “AMAGI-ID-DB”; and 
 iv. Transmitting “AMAGI-WMED-CONTENT” over one or more networks in one or more content formats; 
 
 b. Introducing a delay in the delivery of the network/stored playout content assets in order to identify the watermark and the fingerprint during network reception; 
 c. Performing in the “AMAGI-BARCODE-IN” module the steps of:
 i. Monitoring the network/stored playout content assets at the time of reception to detect “AMAGI-WMED-CONTENT”; 
 ii. Upon detecting previously inserted watermarks (in step a) in “AMAGI-WMED-CONTENT”:
 a) Extracting the corresponding “AMAGI-ID” from the network content asset; 
 b) Extracting corresponding fingerprints for the network content asset; and 
 c) Matching the extracted fingerprints obtained in step (b) with one or more previously stored “AMAGI-FP” in the “AMAGI-ID-DB”; 
 
 iii. Upon finding a match in step (c(ii)c):
 a) Delivering the local content asset at the correct position in the stream of network/stored playout content assets, thereby replacing the network content asset with a suitable local content asset. 
 
 
 
     
     
         12 . A method for seamless content insertion of one or more local content assets, over one or more network/stored playout content assets using an in-band marker of  claim 11  wherein:
 a. One or more watermarking methods using spread-spectrum, amplitude modulation or using quantization of signals, can be used; 
 b. The watermark uses the network content asset as the carrier channel; 
 c. The watermark effects minimal distortion in the audio and video within the network content asset; and 
 d. The watermark is independent of the delivery networks, which might have different compression and/or decompression of the audio-video content, and yet the watermark is reliably transmitted. 
 
     
     
         13 . A method for seamless content insertion of one or more local content assets, over one or more network/stored playout content assets using an in-band marker of  claim 11  wherein:
 a. One or more fingerprinting methods using perceptual characteristics of audio or video streams within one or more network/stored playout content assets, can be used to extract a unique bit pattern that at best uniquely identifies the network/stored playout content assets; and 
 b. Fingerprinting methods used utilize a hash function or a low complexity compression of audio and or video for the purpose of unique identification of the content, rather than near-exact reproduction. 
 
     
     
         14 . A method for seamless content insertion of one or more local content assets, over one or more network/stored playout content assets using an in-band marker of  claim 11   claim 11  wherein if the fingerprint extracted in step g(ii) match with a unique “AMAGI-FP” in “AMAGI-ID-DB”, it is used along with the previously inserted watermark detected to identify the position of insertion of local content assets in the stream of network/stored playout content assets and the steps of watermarking and fingerprinting can be performed interchangably. 
     
     
         15 . A method for seamless content insertion of one or more local content assets, over one or more network/stored playout content assets using an in-band marker of  claim 11  wherein the delay introduced in step 1(e) is determined as the maximum value between:
 a. The duration elapsing during extraction of the audio-video watermark; and 
 b. The duration elapsing during matching the extracted fingerprint. 
 
     
     
         16 . A method for seamless content insertion of one or more local content assets, over one or more network/stored playout content assets using an in-band marker of  claim 11  wherein watermark is extracted from both audio and video of the network content asset or from either one of audio or video to identify the existence of “AMAGI-ID” watermark. 
     
     
         17 . A method for seamless content insertion of one or more local content assets, over one or more network/stored playout content assets using an in-band marker of  claim 11  wherein the fingerprint matching is exercised across −5 to +10 video frames, from the position of identification of the watermark in order to obtain the exact position of the point of insertion. 
     
     
         18 . A method for seamless content insertion of one or more local content assets, over one or more network/stored playout content assets using an in-band marker of  claim 11  wherein one or more fingerprint matching procedures can be used to identify different audio and video insertion points. 
     
     
         19 . A method for seamless content insertion of one or more local content assets, over one or more network/stored playout content assets using an in-band marker of  claim 11  wherein either one of audio or video fingerprint matching and use the point of insertion identified for either audio or video to estimate the point of insertion for both audio and video. 
     
     
         20 . A method for seamless content insertion of one or more local content assets, over one or more network/stored playout content assets using an in-band marker of  claim 11  wherein “AMAGI-BARCODE-IN” can be either of a standalone module or integrated as part of one or more pre-existing content play out, content editing, content post-production or content encoding devices. 
     
     
         21 . A method for seamless content insertion of one or more local content assets, over one or more network/stored playout content assets using an in-band marker of  claim 11  wherein “AMAGI-BARCODE-OUT” can be either of a standalone module or integrated as part of one or more pre-existing content receiver, content insertion, content decoder, or content rendering devices. 
     
     
         22 . A method for seamless content insertion of one or more local content assets, over one or more network/stored playout content assets using an in-band marker of  claim 1  used in Internet content transmission networks by using “AMAGI-BARCODE-IN” on selected internet content assets during content creation, such that “AMAGI-BARCODE-OUT” is used to identify the watermarked internet content asset along with the point of insertion during unicast or multicast streaming to viewers. 
     
     
         23 . A system for seamless content insertion of one or more local content assets, over one or more network/stored playout content assets using an in-band marker used in cable head-ends, set-top boxes, internet browsers and internet content delivery networks using audio-video fingerprinting and water marking with (a) “AMAGI-BARCODE-IN” subsystem, (b) an entertainment content play out server, (c) an ad content play out server, (d) traffic-billing system, (e) one or more encoder or decoder, (f) “AMAGI-BARCODE-OUT” subsystem and (g) cable and satellite TV plants configured to support a plurality of networks, in a plurality of content formats, wherein:
 a. The “AMAGI-BARCODE-IN” subsystem is configured to:
 i. Embed a unique network content asset identifier “AMAGI-ID” as an audio-video watermark, into the network content asset, the watermarked network content asset being labeled “AMAGI-WMED-CONTENT”; 
 ii. Extract a unique audio-video fingerprint “AMAGI-FP” from the network content asset; 
 iii. Store “AMAGI-ID” and “AMAGI-FP” in a data store “AMAGI-ID-DB”; 
 iv. Transmit “AMAGI-WMED-CONTENT” over one or more networks in one or more content formats; 
 
 b. A delay is introduced in the delivery of the network/stored playout content assets in order to identify the watermark and the fingerprint during network reception; 
 c. The “AMAGI-BARCODE-IN” subsystem is configured to:
 i. Monitor the network/stored playout content assets at the time of reception to detect “AMAGI-WMED-CONTENT”; 
 ii. Upon detecting previously inserted watermarks (in step a) in “AMAGI-WMED-CONTENT”:
 a) Extract the corresponding “AMAGI-ID” from the network content asset; 
 b) Extract corresponding fingerprints for the network content asset; and 
 c) Match the extracted fingerprints obtained in step (b) with one or more previously stored “AMAGI-FP” in the “AMAGI-ID-DB”; 
 
 iii. Upon finding a match in (c (ii) c) the local content asset is delivered at the correct position in the stream of network/stored playout content assets, thereby replacing the network content asset with a suitable local content asset. 
 
 
     
     
         24 . A system for seamless content insertion of one or more local content assets, over one or more network/stored playout content assets using an in-band marker of  claim 23  wherein:
 a. One or more watermarking methods using spread-spectrum, amplitude modulation or using quantization of signals, can be used; 
 b. The watermark uses the network content asset as the carrier channel; 
 c. The watermark effects minimal distortion in the audio and video within the network content asset; and 
 d. The watermark is independent of the delivery networks; which might have different compression and/or decompression of the audio-video content, and yet the watermark is reliably transmitted. 
 
     
     
         25 . A system for seamless content insertion of one or more local content assets, over one or more network/stored playout content assets using an in-band marker of  claim 23  wherein:
 a. One or more fingerprinting methods using perceptual characteristics of audio or video streams within one or more network/stored playout content assets, can be used to extract a unique bit pattern that at best uniquely identifies the network/stored playout content assets; and 
 b. Fingerprinting methods used utilize a hash function or a low complexity compression of audio and or video for the purpose of unique identification of the content, rather than near-exact reproduction. 
 
     
     
         26 . A system for seamless content insertion of one or more local content assets, over one or more network/stored playout content assets using an in-band marker of  claim 23   claim 11  wherein if the fingerprint extracted in step g(ii) match with a unique “AMAGI-FP” in “AMAGI-ID-DB”, it is used along with the previously inserted watermark detected to identify the position of insertion of local content assets in the stream of network/stored playout content assets and the steps of watermarking and fingerprinting can be performed interchangably. 
     
     
         27 . A system for seamless content insertion of one or more local content assets, over one or more network/stored playout content assets using an in-band marker of  claim 23  wherein the delay introduced is determined as the maximum value between:
 a. The duration elapsing during extraction of the audio-video watermark; and 
 b. The duration elapsing during matching the extracted fingerprint. 
 
     
     
         28 . A system for seamless content insertion of one or more local content assets, over one or more network/stored playout content assets using an in-band marker of  claim 23  wherein the watermark is extracted from both audio and video of the network content asset or from either one of audio or video to identify the existence of “AMAGI-ID” watermark. 
     
     
         29 . A system for seamless content insertion of one or more local content assets, over one or more network/stored playout content assets using an in-band marker of  claim 23  wherein:
 a. The fingerprint matching is exercised across −5 to +10 video frames, from the position of identification of the watermark in order to obtain the exact position of the point of insertion; 
 b. One or more fingerprint matching procedures are used to identify different audio and video insertion points; and 
 c. Either one of audio or video fingerprint matching and use the point of insertion identified for either audio or video to estimate the point of insertion for both audio and video. 
 
     
     
         30 . A system for seamless content insertion of one or more local content assets, over one or more network/stored playout content assets using an in-band marker of  claim 23  wherein:
 a. The “AMAGI-BARCODE-IN” can be either of a standalone module or integrated as part of one or more pre-existing content play out, content editing, content post-production or content encoding devices; and 
 b. The “AMAGI-BARCODE-OUT” can be either of a standalone module or integrated as part of one or more pre-existing content receiver, content insertion, content decoder, or content rendering devices.