Video analysis, archiving and alerting methods and apparatus for a distributed, modular and extensible video surveillance system

Abstract

Systems and methods are disclosed for analyzing, managing and cataloguing video tracks generated by a video surveillance system. Systems and methods operating in accordance with the present invention receive video tracks from a video surveillance system; analyze them for particular categories of, for example, objects or events, assign unique identifying tags to the video tracks; and save them in highly-cross-referenced databases so that video tracks containing, for example, particular objects or events can be easily searched for and recalled. In addition, the present invention provides browsing facilities for viewing events that have just elapsed, events that are occurring in near real-time.

1 - 97 . (canceled) 
   
   
       98 . A signal-bearing medium tangibly embodying a program of machine-readable instructions executable by a digital processing apparatus of a computer system to perform operations comprising management of video surveillance activities using an electronic video surveillance system, the operations comprising:
 applying a video analysis application tool to video gathered by a video surveillance camera in real time, wherein the video analysis application tool comprises at least one user-specified video analysis criterion, and where application of the video analysis application tool to the video generates a data stream related to the at least one user-specified video analysis criterion;   transmitting the data stream to an electronic database and application system;   storing information comprising the data stream in the electronic database and application system;   selecting at least a portion of the information comprising the data stream for forwarding to a video surveillance analyst performing video surveillance activities, wherein the selection is made based on at least one user-specified information forwarding criterion; and   forwarding the selected information to the video surveillance analyst for display on an interface device of the electronic video surveillance system.   
   
   
       99 . The signal-bearing medium of  claim 98  wherein forwarding the selected information occurs in real time. 
   
   
       100 . The signal-bearing medium of  claim 98  where the user-specified information forwarding criterion comprises an alarm event related to the at least one user-specified video analysis criterion, wherein the alarm event is intended to alert the video surveillance analyst. 
   
   
       101 . The signal-bearing medium of  claim 100  where the user-specified video analysis criterion comprises vehicle speed and where the user-specified information forwarding criterion comprises an alarm triggered when a speed limit is exceeded by a vehicle observed by the video surveillance camera. 
   
   
       102 . The signal-bearing medium of  claim 100  where the user-specified video analysis criterion comprises loitering activity and where the user-specified information forwarding criterion comprises an alarm triggered by the occurrence of loitering activity in a field of view observed by the video surveillance camera. 
   
   
       103 . The signal-bearing medium of  claim 100  where the user-specified video analysis criterion comprises a vehicular accident condition and where the user-specified information forwarding criterion comprises an alarm triggered upon occurrence of a vehicular accident occurring in a field of view monitored by the video surveillance camera. 
   
   
       104 . The signal-bearing medium of  claim 100  where the user-specified video analysis criterion comprises identification of a specific human being and where the user-specified forwarding criterion comprises an alarm triggered when the specific human being is observed in a field of view monitored by the video surveillance camera. 
   
   
       105 . The signal-bearing medium of  claim 100  where the user-specified video analysis criterion comprises the detection of a specific activity and where the user-specified forwarding criterion comprises an alarm triggered when the specific activity occurs in a field of view monitored by the video surveillance camera. 
   
   
       106 . The signal-bearing medium of  claim 98  where the operations further comprise:
 assigning unique identifiers to segments of video gathered by the video surveillance camera; and   transmitting the segments to the electronic database and application system for storage.   
   
   
       107 . The signal-bearing medium of  claim 106  where the data stream comprises at least pointers to video segments satisfying the at least one user-specified video analysis criterion, wherein the pointers comprise video tracks. 
   
   
       108 . The signal-bearing medium of  claim 107  where the operations further comprise:
 forwarding video corresponding to video segments identified by the video track to video surveillance analysts for viewing on a display device of the video surveillance system, wherein the forwarding occurs in dependence on at least one user-specified video forwarding criterion.   
   
   
       109 . The signal-bearing medium of  claim 108  where forwarding the video corresponding to video segments identified by video track occurs in real time. 
   
   
       110 . The signal-bearing medium of  claim 108  where the user-specified video forwarding criterion comprises an alarm event related to the at least one user-specified video analysis criterion, wherein the alarm event is intended to alert the video surveillance analyst. 
   
   
       111 . The signal-bearing medium of  claim 108  where the operations further comprise:
 generating at least one keyframe each for at least a portion of the video segments identified by video tracks;   associating each keyframe with its respective video segment through a unique identifier; and   saving the keyframe to the electronic database and application system.   
   
   
       112 . The signal-bearing medium of  claim 111  where the operations further comprise:
 selecting at least one keyframe for forwarding to a video surveillance analyst performing video surveillance activities, wherein the selection is made based on at least one user-specified keyframe forwarding criterion; and   forwarding the selected keyframe to the video surveillance analyst for display on a display device of the electronic video surveillance system.   
   
   
       113 . The signal-bearing medium of  claim 112  where forwarding the at least one keyframe occurs in real time. 
   
   
       114 . The signal-bearing medium of  claim 112  where the user-specified keyframe forwarding criterion comprises an alarm event related to the at least one user-specified video analysis criterion, wherein the alarm event is intended to alert the video surveillance analyst. 
   
   
       115 . The signal-bearing medium of  claim 112  where the operations further comprise:
 receiving a request for the video segment corresponding to the keyframe from the video surveillance analyst;   retrieving the video segment corresponding to the keyframe from the electronic database and application system using the track; and   transmitting the video segment to a display device of the video surveillance system for viewing by the video surveillance analyst.   
   
   
       116 . The signal-bearing medium of  claim 111  where the operations further comprise:
 selecting a plurality of keyframes for forwarding to a display device of the video surveillance system for viewing by a video surveillance analyst;   forwarding the plurality of keyframes to the display device; and   simultaneously displaying the plurality of keyframes on the display device.   
   
   
       117 . The signal-bearing medium of  claim 116  where the operations further comprise:
 receiving a request for the video segment corresponding to at least one of the keyframes from the video surveillance analyst;   retrieving the video segment corresponding to the at least one keyframe from the electronic database and application system using the track; and   transmitting the video segment for display on the display device of the video surveillance system for viewing by the video surveillance analyst.   
   
   
       118 . The signal-bearing medium of  claim 98  where the operations further comprise:
 receiving a query for information stored in the electronic database and application system;   retrieving information responsive to the query from the electronic database and application system; and   forwarding the information responsive to the query to a display device of the video surveillance system for viewing by a video surveillance analyst.   
   
   
       119 . The signal-bearing medium of  claim 107  where the operations further comprise:
 receiving a query for information stored in the electronic database and application system, wherein the query comprises at least a request for video satisfying a user-specified selection criterion;   retrieving information responsive to the query from the electronic database and application system, wherein the information comprises video track;   retrieving the video segments corresponding to the video track from the electronic database and application system; and   transmitting the video segments to a display device of the video surveillance system for viewing by a video surveillance analyst.   
   
   
       120 . The signal-bearing medium of  claim 111  where the operations further comprise:
 receiving a query for information stored in the electronic database and application system, wherein the query comprises at least a request for keyframes satisfying a user-specified selection criterion;   retrieving information responsive to the query from the electronic database and application system, wherein the information comprises keyframe identification data;   retrieving the keyframes corresponding to the keyframe identification data from the electronic database and application system; and   transmitting the keyframes to a display device of the video surveillance system for viewing by a video surveillance analyst.   
   
   
       121 . The signal-bearing medium of  claim 119  wherein the operations further comprise:
 receiving a request for video segments corresponding to the keyframes from the video surveillance analyst;   retrieving the video segments corresponding to the keyframes from the electronic database and application system using video tracks; and   transmitting the video segments for display on the display device of the video surveillance system for viewing by the video surveillance analyst.   
   
   
       122 . The video surveillance system of claim  52  where the alert criterion is specified based on statistics derived from data generated by the video analysis engine. 
   
   
       123 . The video surveillance system of claim  61  where the video surveillance system comprises a plurality of video surveillance system modules, wherein each of the video surveillance system modules further comprises a video surveillance camera having the multiple view programming feature, the video surveillance system further comprising:
 a cluster specification system to specify a cluster of views, the cluster of views corresponding to at least two views selected from at least two of the video surveillance cameras comprising the video surveillance system modules.   
   
   
       124 . The video surveillance system of  claim 123  further comprising:
 a cluster video analysis criterion specification system to specify a cluster video analysis criterion to be applied to a cluster of views specified with the cluster specification system, wherein the video analysis will be performed by the video analysis engines of the video surveillance system modules.   
   
   
       125 . The video surveillance system of  claim 124  further comprising:
 a cluster alert criterion specification system to specify an alert criterion related to a video analysis criterion applied to video gathered by the video surveillance cameras comprising the cluster of views, wherein the video analysis criterion is specified with the cluster video analysis criterion specification system.   
   
   
       126 . The video surveillance system of  claim 124  where the video analysis engines perform video analysis based on a cluster video analysis criterion, and generate video analysis data for the cluster of views in dependence on the video analysis criterion. 
   
   
       127 . The video surveillance system of  claim 126  where the at least one database and application system saves the video analysis data generated by the video analysis engines. 
   
   
       128 . The video surveillance system of  claim 127  where the at least one database and application system receives queries for video analysis data saved in the at least one database and application system, wherein the video analysis data was generated by the video analysis engines applying the cluster video analysis criterion.