Scalable, adaptable, and manageable system for multimedia identification

Abstract

An architecture for a multimedia search system is described. To perform similarity matching of multimedia query frames against reference content, reference database comprising of a cluster index using cluster keys to perform similarity matching and a multimedia index to perform sequence matching is built. Methods to update and maintain the reference database that enables addition and removal of the multimedia contents, including portions of multimedia content, from the reference database in a running system are described. Hierarchical multi-level partitioning methods to organize the reference database are presented. Smart partitioning of the reference multimedia content according to the nature of the multimedia content, and according to the popularity among the social media, that supports scalable fast multimedia identification is also presented. A caching mechanism for multimedia search queries in a centralized or in a decentralized distributed system and a client based local multimedia search system enabling multimedia tracking are described.

We claim: 
     
       1. A method of tracking multimedia content, the method comprising:
 searching in a remote database in response to a query sent from a user device for multimedia content of a multimedia program that is currently playing on the user device; 
 finding matching signatures for the multimedia content in a reference program stored in the remote database; 
 retrieving the matching signatures and reference signatures for succeeding segments of the reference program beginning from the point in the reference program the multimedia content was found in the remote database; 
 storing the matching signatures and reference signatures in a local database on the user device; 
 tracking the multimedia program in the local database to find matching content for the succeeding segments of the reference program; 
 determining the multimedia content has changed to different multimedia content than that stored in the succeeding segments of the reference program; and 
 replacing the determined different multimedia content to a preselected multimedia content. 
 
     
     
       2. The method of  claim 1  further comprising:
 playing the preselected multimedia content for a prespecified duration; and 
 returning to play the multimedia program upon detecting matching content for subsequent segments of the reference program. 
 
     
     
       3. The method of  claim 1 , wherein the preselected multimedia content is a new source of multimedia content. 
     
     
       4. The method of  claim 3 , wherein the new source of multimedia content is a picture-in-picture mode of operation with another channel. 
     
     
       5. The method of  claim 1 , wherein the preselected multimedia content is a reduction in the audio on the user device. 
     
     
       6. The method of  claim 1 , wherein the succeeding segments of the reference program is a full reference program including the matching signatures from the point in the reference program the multimedia content was found in the remote database. 
     
     
       7. The method of  claim 1  further comprising:
 blocking the playing of the preselected multimedia content; and 
 returning to display the multimedia program upon detecting matching content for subsequent segments of the reference program. 
 
     
     
       8. The method of  claim 1 , wherein the searching in the remote database uses a similarity search method combined with a correlation search method. 
     
     
       9. The method of  claim 1 , wherein the tracking to find matching content uses a correlation method. 
     
     
       10. An apparatus for tracking multimedia content, the apparatus comprising:
 a user processing device configured for sending a query to a remote database to search for multimedia content of a multimedia program identified by a user, wherein matching signatures are found for the multimedia content in a reference program stored in the remote database, and the matching signatures and reference signatures for succeeding segments of the reference program beginning from the point in the reference program the multimedia content was found are retrieved from the remote database; and 
 a local database coupled to the user processing device and configured for storing the matching signatures and reference signatures retrieved from the remote database, 
 wherein the user processing device is further configured to track the multimedia program in the local database to find matching content for the succeeding segments of the reference program, 
 the user processing device is further configured to determine the multimedia content has changed to different multimedia content than that stored in the succeeding segments of the reference program, and 
 wherein the user processing device is further configured to replace the determined different multimedia content to a preselected multimedia content. 
 
     
     
       11. The apparatus of  claim 10 , wherein the local database is stored in a memory internal to the user processing device. 
     
     
       12. The apparatus of  claim 10 , wherein the local database is stored in a random access memory and a hard disk drive internal to the user processing device. 
     
     
       13. The apparatus of  claim 10 , wherein each reference signature includes a cluster key comprising a link reference address to a list of signature records stored in the local database. 
     
     
       14. The apparatus of  claim 13 , wherein the cluster key is generated by applying a fingerprinting process to bits of a reference fingerprint having a first number of bits, wherein the cluster key comprises a second number of bits that is less than the first number of bits. 
     
     
       15. A computer readable non-transitory medium storing a computer program which causes a computer system to perform a method of tracking multimedia content, the method comprising:
 searching in a remote database in response to a query sent from a user device for multimedia content of a multimedia program that is currently playing on the user device; 
 finding matching signatures for the multimedia content in a reference program stored in the remote database; 
 retrieving the matching signatures and reference signatures for succeeding segments of the reference program beginning from the point in the reference program the multimedia content was found in the remote database; 
 storing the matching signatures and reference signatures in a local database on the user device; 
 tracking the multimedia program in the local database to find matching content for the succeeding segments of the reference program; 
 determining the multimedia content has changed to different multimedia content than that stored in the succeeding segments of the reference program; and 
 replacing the determined different multimedia content to a preselected multimedia content. 
 
     
     
       16. The computer readable non-transitory medium method of  claim 15  further comprising:
 playing the preselected multimedia content for a prespecified duration; and 
 returning to play the multimedia program upon detecting matching content for subsequent segments of the reference program. 
 
     
     
       17. The computer readable non-transitory medium method of  claim 15 , wherein the preselected multimedia content is a picture-in-picture mode of operation with another channel. 
     
     
       18. The computer readable non-transitory medium method of  claim 15  further comprising:
 blocking the playing of the preselected multimedia content; and 
 returning to display the multimedia program upon detecting matching content for subsequent segments of the reference program. 
 
     
     
       19. The computer readable non-transitory medium method of  claim 15 , wherein the searching in the remote database uses a similarity search method combined with a correlation search method. 
     
     
       20. The computer readable non-transitory medium method of  claim 15 , wherein each reference signature includes a cluster key comprising a link reference address to a list of signature records stored in the local database. 
     
     
       21. A method comprising:
 generating, by a client device, a first query signature representing a portion of a first content unit, wherein the portion of the first content unit is presented by the client device;   sending, by the client device, to a server device, the generated first query signature;   responsive to sending the generated first query signature, receiving, by the client device, from the server device, a set of reference signatures representing additional portions of the first content unit, wherein the set of reference signatures was selected by the server device based at least in part on the generated first query signature;   generating, by the client device, a second query signature representing a portion of a second content unit that is different from the first content unit;   comparing, by the client device, the generated second query signature with the received set of reference signatures;   determining, by the client device, that the comparing resulted in no match being found; and   responsive to determining that the comparing resulted in no match being found, performing, by the client device, an action.    
     
     
       22. The method of claim 21, wherein the first content unit comprises video content.  
     
     
       23. The method of claim 22, wherein the first content unit further comprises audio content.  
     
     
       24. The method of claim 21, wherein the client device is a television set and wherein presented by the client device comprises displayed by the client device.  
     
     
       25. The method of claim 21, wherein the portion of the second content unit is received but not presented by the client device.  
     
     
       26. The method of claim 21, wherein the first content unit is a television show and the second content unit is an advertisement.  
     
     
       27. The method of claim 21, wherein comparing the generated second query signature with the received set of reference signatures comprises using a track search technique to compare the generated second query signature with the received set of reference signatures.  
     
     
       28. The method of claim 21, wherein comparing the generated second query signature with the received set of reference signatures comprises using a full search query technique to compare the generated second query signature with the received set of reference signatures.  
     
     
       29. The method of claim 21, wherein comparing the generated second query signature with the received set of reference signatures comprises using both a track search technique and a full search query technique to compare the generated second query signature with the received set of reference signatures.  
     
     
       30. The method of claim 21, wherein performing the action comprises:
 using user-preference data associated with a user of the client device to identify a user-preferred action; and   performing the identified user-preferred action.    
     
     
       31. The method of claim 21, wherein performing the action comprises at least temporarily blocking presentation of the second content unit.  
     
     
       32. The method of claim 21, wherein performing the action comprises blocking presentation of the second content unit for a duration of the second content unit.  
     
     
       33. The method of claim 21, wherein performing the action comprises switching to a different channel on which to receive and present content.  
     
     
       34. The method of claim 21, wherein performing the action comprises switching to presenting different content.  
     
     
       35. The method of claim 21, wherein the client device is a television set and wherein performing the action comprises initiating a picture-in-picture mode of the television set.  
     
     
       36. The method of claim 21, wherein performing the action comprises reducing an audio volume level of the client device.  
     
     
       37. A non-transitory computer-readable storage medium having stored thereon program instructions that, when executed by one or more processors of a client device, cause the client device to perform operations comprising:
 generating a first query signature representing a portion of a first content unit, wherein the portion of the first content unit is presented by the client device;   sending, to a server device, the generated first query signature;   responsive to sending the generated first query signature, receiving, from the server device, a set of reference signatures representing additional portions of the first content unit, wherein the set of reference signatures was selected by the server device based at least in part on the generated first query signature;   generating a second query signature representing a portion of a second content unit that is different from the first content unit;   comparing the generated second query signature with the received set of reference signatures;   determining that the comparing resulted in no match being found; and   responsive to determining that the comparing resulted in no match being found, performing an action.    
     
     
       38. The non-transitory computer-readable storage medium of claim 37, wherein the first content unit comprises video content.  
     
     
       39. The non-transitory computer-readable storage medium of claim 38, wherein the first content unit further comprises audio content.  
     
     
       40. The non-transitory computer-readable storage medium of claim 37, wherein the client device is a television set and wherein presented by the client device comprises displayed by the client device.  
     
     
       41. The non-transitory computer-readable storage medium of claim 37, wherein the portion of the second content unit is received but not presented by the client device.  
     
     
       42. The non-transitory computer-readable storage medium of claim 37, wherein the first content unit is a television show and the second content unit is an advertisement.  
     
     
       43. The non-transitory computer-readable storage medium of claim 37, wherein comparing the generated second query signature with the received set of reference signatures comprises using a track search technique to compare the generated second query signature with the received set of reference signatures.  
     
     
       44. The non-transitory computer-readable storage medium of claim 37, wherein comparing the generated second query signature with the received set of reference signatures comprises using a full search query technique to compare the generated second query signature with the received set of reference signatures.  
     
     
       45. The non-transitory computer-readable storage medium of claim 37, wherein comparing the generated second query signature with the received set of reference signatures comprises using both a track search technique and a full search query technique to compare the generated second query signature with the received set of reference signatures.  
     
     
       46. The non-transitory computer-readable storage medium of claim 37, wherein performing the action comprises:
 using user-preference data associated with a user of the client device to identify a user-preferred action; and   performing the identified user-preferred action.    
     
     
       47. The non-transitory computer-readable storage medium of claim 37, wherein performing the action comprises at least temporarily blocking presentation of the second content unit.  
     
     
       48. The non-transitory computer-readable storage medium of claim 37, wherein performing the action comprises blocking presentation of the second content unit for a duration of the second content unit.  
     
     
       49. The non-transitory computer-readable storage medium of claim 37, wherein performing the action comprises switching to a different channel on which to receive and present content.  
     
     
       50. The non-transitory computer-readable storage medium of claim 37, wherein performing the action comprises switching to presenting different content.  
     
     
       51. The non-transitory computer-readable storage medium of claim 37, wherein the client device is a television set and wherein performing the action comprises initiating a picture-in-picture mode of the television set.  
     
     
       52. The non-transitory computer-readable storage medium of claim 37, wherein performing the action comprises reducing an audio volume level of the client device.  
     
     
       53. A client device comprising one or more processors and a non-transitory computer-readable storage medium having stored thereon program instructions that, when executed by the one or more processors, cause the client device to perform operations comprising:
 generating a first query signature representing a portion of a first content unit, wherein the portion of the first content unit is presented by the client device;   sending, to a server device, the generated first query signature;   responsive to sending the generated first query signature, receiving, from the server device, a set of reference signatures representing additional portions of the first content unit, wherein the set of reference signatures was selected by the server device based at least in part on the generated first query signature;   generating a second query signature representing a portion of a second content unit that is different from the first content unit;   comparing the generated second query signature with the received set of reference signatures;   determining that the comparing resulted in no match being found; and   responsive to determining that the comparing resulted in no match being found, performing an action.    
     
     
       54. The client device of claim 53, wherein the first content unit comprises video content.  
     
     
       55. The client device of claim 54, wherein the first content unit further comprises audio content.  
     
     
       56. The client device of claim 53, wherein the client device is a television set and wherein presented by the client device comprises displayed by the client device.  
     
     
       57. The client device of claim 53, wherein the portion of the second content unit is received but not presented by the client device.  
     
     
       58. The client device of claim 53, wherein the first content unit is a television show and the second content unit is an advertisement.  
     
     
       59. The client device of claim 53, wherein comparing the generated second query signature with the received set of reference signatures comprises using a track search technique to compare the generated second query signature with the received set of reference signatures.  
     
     
       60. The client device of claim 53, wherein comparing the generated second query signature with the received set of reference signatures comprises using a full search query technique to compare the generated second query signature with the received set of reference signatures.  
     
     
       61. The client device of claim 53, wherein comparing the generated second query signature with the received set of reference signatures comprises using both a track search technique and a full search query technique to compare the generated second query signature with the received set of reference signatures.  
     
     
       62. The client device of claim 53, wherein performing the action comprises:
 using user-preference data associated with a user of the client device to identify a user-preferred action; and   performing the identified user-preferred action.    
     
     
       63. The client device of claim 53, wherein performing the action comprises at least temporarily blocking presentation of the second content unit.  
     
     
       64. The client device of claim 53, wherein performing the action comprises blocking presentation of the second content unit for a duration of the second content unit.  
     
     
       65. The client device of claim 53, wherein performing the action comprises switching to a different channel on which to receive and present content.  
     
     
       66. The client device of claim 53, wherein performing the action comprises switching to presenting different content.  
     
     
       67. The client device of claim 53, wherein the client device is a television set and wherein performing the action comprises initiating a picture-in-picture mode of the television set.  
     
     
       68. The client device of claim 53, wherein performing the action comprises reducing an audio volume level of the client device.  
     
     
       69. A method comprising:
 receiving, by a server device, from a client device, a first query signature that was generated by the client device and that represents a portion of a first content unit, wherein the client device presented the portion of the first content unit;   based at least in part on the received first query signature, selecting a set of reference signatures representing additional portions of the first content unit; and   sending, by the server device, to the client device, the selected set of reference signatures to facilitate the client device:
 (i) comparing a second query signature generated by the client device and that represents a portion of a second content unit that is different from the first content unit with the set of reference signatures; 
 (ii) determining that the comparing resulted in no match being found, and 
 (iii) responsive to determining that the comparing resulted in no match being found, performing an action.  
   
     
     
       70. The method of claim 69, wherein the first content unit comprises video content.  
     
     
       71. The method of claim 70, wherein the first content unit further comprises audio content.  
     
     
       72. The method of claim 69, wherein the client device is a television set and wherein presented the portion of the first content unit comprises displayed the portion of the first content unit.  
     
     
       73. The method of claim 69, wherein the portion of the second content unit is received but not presented by the client device.  
     
     
       74. The method of claim 69, wherein the first content unit is a television show and the second content unit is an advertisement.  
     
     
       75. The method of claim 69, wherein comparing the second query signature with the set of reference signatures comprises using a track search technique to compare the second query signature with the set of reference signatures.  
     
     
       76. The method of claim 69, wherein comparing the second query signature with the set of reference signatures comprises using a full search query technique to compare the second query signature with the set of reference signatures.  
     
     
       77. The method of claim 69, wherein comparing the second query signature with the set of reference signatures comprises using both a track search technique and a full search query technique to compare the second query signature with the set of reference signatures.  
     
     
       78. The method of claim 69, wherein performing the action comprises:
 using user-preference data associated with a user of the client device to identify a user-preferred action; and   performing the identified user-preferred action.    
     
     
       79. The method of claim 69, wherein performing the action comprises at least temporarily blocking presentation of the second content unit.  
     
     
       80. The method of claim 69, wherein performing the action comprises blocking presentation of the second content unit for a duration of the second content unit.  
     
     
       81. The method of claim 69, wherein performing the action comprises switching to a different channel on which to receive and present content.  
     
     
       82. The method of claim 69, wherein performing the action comprises switching to presenting different content.  
     
     
       83. The method of claim 69, wherein the client device is a television set and wherein performing the action comprises initiating a picture-in-picture mode of the television set.  
     
     
       84. The method of claim 69, wherein performing the action comprises reducing an audio volume level of the client device.  
     
     
       85. A non-transitory computer-readable storage medium having stored thereon program instructions that, when executed by one or more processors of a server device, cause the server device to perform operations comprising:
 receiving, from a client device, a first query signature that was generated by the client device and that represents a portion of a first content unit, wherein the client device presented the portion of the first content unit;   based at least in part on the received first query signature, selecting a set of reference signatures representing additional portions of the first content unit; and   sending, to the client device, the selected set of reference signatures to facilitate the client device:
 (i) comparing a second query signature generated by the client device and that represents a portion of a second content unit that is different from the first content unit with the set of reference signatures; 
 (ii) determining that the comparing resulted in no match being found, and 
 (iii) responsive to determining that the comparing resulted in no match being found, performing an action.  
   
     
     
       86. The non-transitory computer-readable storage medium of claim 85, wherein the first content unit comprises video content.  
     
     
       87. The non-transitory computer-readable storage medium of claim 86, wherein the first content unit further comprises audio content.  
     
     
       88. The non-transitory computer-readable storage medium of claim 85, wherein the client device is a television set and wherein presented the portion of the first content unit comprises displayed the portion of the first content unit.  
     
     
       89. The non-transitory computer-readable storage medium of claim 85, wherein the portion of the second content unit is received but not presented by the client device.  
     
     
       90. The non-transitory computer-readable storage medium of claim 85, wherein the first content unit is a television show and the second content unit is an advertisement.  
     
     
       91. The non-transitory computer-readable storage medium of claim 85, wherein comparing the second query signature with the set of reference signatures comprises using a track search technique to compare the second query signature with the set of reference signatures.  
     
     
       92. The non-transitory computer-readable storage medium of claim 85, wherein comparing the second query signature with the set of reference signatures comprises using a full search query technique to compare the second query signature with the set of reference signatures.  
     
     
       93. The non-transitory computer-readable storage medium of claim 85, wherein comparing the second query signature with the set of reference signatures comprises using both a track search technique and a full search query technique to compare the second query signature with the set of reference signatures.  
     
     
       94. The non-transitory computer-readable storage medium of claim 85, wherein performing the action comprises:
 using user-preference data associated with a user of the client device to identify a user-preferred action; and   performing the identified user-preferred action.    
     
     
       95. The non-transitory computer-readable storage medium of claim 85, wherein performing the action comprises at least temporarily blocking presentation of the second content unit.  
     
     
       96. The non-transitory computer-readable storage medium of claim 85, wherein performing the action comprises blocking presentation of the second content unit for a duration of the second content unit.  
     
     
       97. The non-transitory computer-readable storage medium of claim 85, wherein performing the action comprises switching to a different channel on which to receive and present content.  
     
     
       98. The non-transitory computer-readable storage medium of claim 85, wherein performing the action comprises switching to presenting different content.  
     
     
       99. The non-transitory computer-readable storage medium of claim 85, wherein the client device is a television set and wherein performing the action comprises initiating a picture-in-picture mode of the television set.  
     
     
       100. The non-transitory computer-readable storage medium of claim 85, wherein performing the action comprises reducing an audio volume level of the client device.  
     
     
       101. A server device comprising one or more processors and a non-transitory computer-readable storage medium having stored thereon program instructions that, when executed by the one or more processors, cause the server device to perform operations comprising:
 receiving, from a client device, a first query signature that was generated by the client device and that represents a portion of a first content unit, wherein the client device presented the portion of the first content unit;   based at least in part on the received first query signature, selecting a set of reference signatures representing additional portions of the first content unit; and   sending, to the client device, the selected set of reference signatures to facilitate the client device:
 (i) comparing a second query signature generated by the client device and that represents a portion of a second content unit that is different from the first content unit with the set of reference signatures; 
 (ii) determining that the comparing resulted in no match being found, and 
 (iii) responsive to determining that the comparing resulted in no match being found, performing an action.  
   
     
     
       102. The server device of claim 101, wherein the first content unit comprises video content.  
     
     
       103. The server device of claim 102, wherein the first content unit further comprises audio content.  
     
     
       104. The server device of claim 101, wherein the client device is a television set and wherein presented the portion of the first content unit comprises displayed the portion of the first content unit.  
     
     
       105. The server device of claim 101, wherein the portion of the second content unit is received but not presented by the client device.  
     
     
       106. The server device of claim 101, wherein the first content unit is a television show and the second content unit is an advertisement.  
     
     
       107. The server device of claim 101, wherein comparing the second query signature with the set of reference signatures comprises using a track search technique to compare the second query signature with the set of reference signatures.  
     
     
       108. The server device of claim 101, wherein comparing the second query signature with the set of reference signatures comprises using a full search query technique to compare the second query signature with the set of reference signatures.  
     
     
       109. The server device of claim 101, wherein comparing the second query signature with the set of reference signatures comprises using both a track search technique and a full search query technique to compare the second query signature with the set of reference signatures.  
     
     
       110. The server device of claim 101, wherein performing the action comprises:
 using user-preference data associated with a user of the client device to identify a user-preferred action; and   performing the identified user-preferred action.    
     
     
       111. The server device of claim 101, wherein performing the action comprises at least temporarily blocking presentation of the second content unit.  
     
     
       112. The server device of claim 101, wherein performing the action comprises blocking presentation of the second content unit for a duration of the second content unit.  
     
     
       113. The server device of claim 101, wherein performing the action comprises switching to a different channel on which to receive and present content.  
     
     
       114. The server device of claim 101, wherein performing the action comprises switching to presenting different content.  
     
     
       115. The server device of claim 101, wherein the client device is a television set and wherein performing the action comprises initiating a picture-in-picture mode of the television set.  
     
     
       116. The server device of claim 101, wherein performing the action comprises reducing an audio volume level of the client device.