Time-series based analytics using video streams

Abstract

Methods and systems for detecting and predicting anomalies include processing frames of a video stream to determine values of a feature corresponding to each frame. A feature time series is generated that corresponds to values of the identified feature over time. A matrix profile is generated that identifies similarities of sub-sequences of the time series to other sub-sequences of the feature time series. An anomaly is detected by determining that a value of the matrix profile exceeds a threshold value. An automatic action is performed responsive to the detected anomaly.

What is claimed is: 
     
       1. A method of detecting and responding to an anomaly, comprising:
 processing frames of a video stream to determine values of a feature corresponding to each frame; 
 generating a feature time series that corresponds to values of the identified feature over time; 
 determining a matrix profile that identifies similarities of sub-sequences of the time series to other sub-sequences of the feature time series, wherein the matrix profile includes a matrix having elements that each represent a similarity between a respective pair of sub-sequences of the time series and further includes a profile time series having values that correspond to each sub-sequence of the feature time series and that are determined by the most similar other sub-sequence of the feature time series; 
 detecting an anomaly by determining that a value of the matrix profile exceeds a threshold value; and 
 performing an automatic action responsive to the detected anomaly. 
 
     
     
       2. The method of  claim 1 , wherein processing the frames of the video stream includes performing object detection. 
     
     
       3. The method of  claim 2 , wherein the feature time series includes a number of detected objects at each frame. 
     
     
       4. The method of  claim 1 , updating the feature time series and the matrix profile in response to newly received frames of the video stream. 
     
     
       5. The method of  claim 1 , wherein the automatic action includes an action selected from the group consisting of sounding an alarm, locking doors, engaging or disengaging traffic control devices, changing the operational status of one or more machines, triggering an environmental control, and triggering a fire suppression system. 
     
     
       6. A method of predicting an anomaly, comprising:
 generating a time series, based on frames of a video stream; 
 determining a matrix profile, based on the time series, that identifies similarities of sub-sequences of the time series to other sub-sequences of the time series, wherein the matrix profile includes a matrix having elements that each represent a similarity between a respective pair of sub-sequences of the time series and further includes a profile time series having values that correspond to each sub-sequence of the feature time series and that are determined by the most similar other sub-sequence of the feature time series; 
 detecting an anomaly by determining that a value of the matrix profile exceeds a threshold value; 
 identifying an anomaly precursor that includes a pattern from the time series occurring before the detected anomaly; 
 detecting the anomaly precursor in a newly received sub-sequence of the time series to predict an anomaly, by matching the identified anomaly precursor to the newly received sub-sequence; and 
 performing an automatic action to prevent the predicted anomaly. 
 
     
     
       7. The method of  claim 6 , wherein the automatic action includes an action selected from the group consisting of sounding an alarm, locking doors, engaging or disengaging traffic control devices, changing the operational status of one or more machines, triggering an environmental control, and triggering a fire suppression system. 
     
     
       8. The method of  claim 6 , further comprising saving the identified anomaly in a precursor database. 
     
     
       9. The method of  claim 8 , further comprising generating the newly generated sub-sequence from new time series information, wherein matching the identified anomaly precursor to the newly received sub-sequence includes determining a similarity between the newly generated sub-sequence to each identified anomaly in the precursor database. 
     
     
       10. A system for detecting and responding to an anomaly, comprising:
 a hardware processor; 
 a memory that stores a computer program product, which, when executed by the hardware processor, causes the hardware processor to:
 process frames of a video stream to determine values of a feature corresponding to each frame; 
 generate a feature time series that corresponds to values of the identified feature over time; 
 determine a matrix profile that identifies similarities of sub-sequences of the time series to other sub-sequences of the feature time series, wherein the matrix profile includes a matrix having elements that each represent a similarity between a respective pair of sub-sequences of the time series and further includes a profile time series having values that correspond to each sub-sequence of the feature time series and that are determined by the most similar other sub-sequence of the feature time series; 
 detect an anomaly by determining that a value of the matrix profile exceeds a threshold value; and 
 perform an automatic action responsive to the detected anomaly. 
 
 
     
     
       11. The system of  claim 10 , wherein the computer program product further causes the hardware processor to perform object detection. 
     
     
       12. The system of  claim 11 , wherein the feature time series includes a number of detected objects at each frame. 
     
     
       13. The system of  claim 10 , wherein the computer program product further causes the hardware processor to update the feature time series and the matrix profile in response to newly received frames of the video stream. 
     
     
       14. The system of  claim 10 , wherein the automatic action includes an action selected from the group consisting of sounding an alarm, locking doors, engaging or disengaging traffic control devices, changing the operational status of one or more machines, triggering an environmental control, and triggering a fire suppression system.