US12001513B2ActiveUtilityA1
Self-optimizing video analytics pipelines
Est. expiryNov 30, 2040(~14.4 yrs left)· nominal 20-yr term from priority
G06N 3/09G06N 3/092G06N 3/0464G06F 18/217G06F 9/5027G06N 3/08G06V 10/94G06V 20/46G06V 20/52G06V 10/82G06F 9/5055G06F 2209/508G06N 3/006
62
PatentIndex Score
0
Cited by
12
References
20
Claims
Abstract
A method for implementing a self-optimized video analytics pipeline is presented. The method includes decoding video files into a sequence of frames, extracting features of objects from one or more frames of the sequence of frames of the video files, employing an adaptive resource allocation component based on reinforcement learning (RL) to dynamically balance resource usage of different microservices included in the video analytics pipeline, employing an adaptive microservice parameter tuning component to balance accuracy and performance of a microservice of the different microservices, applying a graph-based filter to minimize redundant computations across the one or more frames of the sequence of frames, and applying a deep-learning-based filter to remove unnecessary computations resulting from mismatches between the different microservices in the video analytics pipeline.
Claims
exact text as granted — not AI-modifiedWhat is claimed is:
1. A method for implementing a self-optimized video analytics pipeline, the method comprising:
decoding video files into a sequence of frames;
extracting features of objects from one or more frames of the sequence of frames of the video files;
employing an adaptive resource allocation component based on reinforcement learning (RL) to dynamically balance resource usage of different microservices included in the video analytics pipeline;
employing an adaptive microservice parameter tuning component to balance accuracy and performance of a microservice of the different microservices;
applying a graph-based filter to minimize redundant computations across the one or more frames of the sequence of frames; and
applying a deep-learning-based filter to remove unnecessary computations resulting from mismatches between the different microservices in the video analytics pipeline.
2. The method of claim 1 , wherein a state in the adaptive resource allocation component is a vector including at least backlog of input queue of each microservice, current utilization of system resources, a fraction of each system resource assigned to each of the microservices, and metrics derived from content of the video files.
3. The method of claim 1 , wherein a microservice of the different microservices includes a monolithic application having functions that are dis-aggregated into separate object detection and feature extraction microservices.
4. The method of claim 1 , wherein the adaptive microservice parameter tuning component tunes a batch size parameter and an image resolution parameter by RL.
5. The method of claim 1 , wherein the graph-based filter removes repeated objects from consecutive frames within the sequence of frames by creating a graph G=(V, E), where each object forms a vertex v i and an edge (v i , v j ) between two vertices v i and v j , if and only if, these objects are close to each other, the closeness of two objects determined by spatial closeness and temporal closeness used to compute a distance metric.
6. The method of claim 5 , wherein the spatial closeness employs an overlap ratio between bounding boxes of the objects from the consecutive frames, the overlap ratio given as:
overlap
(
v
i
,
v
j
)
=
area
(
v
i
⋂
v
j
)
min
(
area
(
v
i
)
,
area
(
v
j
)
)
.
7. The method of claim 5 , wherein the temporal closeness employs a lag between the objects from the consecutive frames, the lag given as:
lag
(
v
i
,
v
j
)
=
frame
#
v
i
-
frame
#
v
j
frameInBatch
.
8. The method of claim 5 , wherein the graph is partitioned into multiple sub-graphs, each sub-graph is flattened into a list of objects, and duplicates are labeled in a flattened connected component.
9. A non-transitory computer-readable storage medium comprising a computer-readable program for implementing a self-optimized video analytics pipeline, wherein the computer-readable program when executed on a computer causes the computer to perform the steps of;
decoding video files into a sequence of frames;
extracting features of objects from one or more frames of the sequence of frames of the video files;
employing an adaptive resource allocation component based on reinforcement learning (RL) to dynamically balance resource usage of different microservices included in the video analytics pipeline;
employing an adaptive microservice parameter tuning component to balance accuracy and performance of a microservice of the different microservices;
applying a graph-based filter to minimize redundant computations across the one or more frames of the sequence of frames; and
applying a deep-learning-based filter to remove unnecessary computations resulting from mismatches between the different microservices in the video analytics.
10. The non-transitory computer-readable storage medium of claim 9 , wherein a state in the adaptive resource allocation component is a vector including at least backlog of input queue of each microservice, current utilization of system resources, a fraction of each system resource assigned to each of the microservices, and metrics derived from content of the video files.
11. The non-transitory computer-readable storage medium of claim 9 , wherein a microservice of the different microservices includes a monolithic application having functions that are dis-aggregated into separate object detection and feature extraction microservices.
12. The non-transitory computer-readable storage medium of claim 9 , wherein the adaptive microservice parameter tuning component tunes a batch size parameter and an image resolution parameter by RL.
13. The non-transitory computer-readable storage medium of claim 9 , wherein the graph-based filter removes repeated objects from consecutive frames within the sequence of frames by creating a graph G=(V, E), where each object forms a vertex v i and an edge e(v i , v j ) between two vertices v i and v j , if and only if, these objects arc close to each other, the closeness of two objects determined by spatial closeness and temporal closeness used to compute a distance metric.
14. The non-transitory computer-readable storage medium of claim 13 , wherein the spatial closeness employs an overlap ratio between bounding boxes of the objects from the consecutive frames, the overlap ratio given as:
overlap
(
v
i
,
v
j
)
=
area
(
v
i
⋂
v
j
)
min
(
area
(
v
i
)
,
area
(
v
j
)
)
.
15. The non-transitory computer-readable storage medium of claim 13 , wherein the temporal closeness employs a lag between the objects from the consecutive frames, the lag given as:
lag
(
v
i
,
v
j
)
=
frame
#
v
i
-
frame
#
v
j
frameInBatch
.
16. The non-transitory computer-readable storage medium of claim 13 , wherein the graph is partitioned into multiple sub-graphs, each sub-graph is flattened into a list of objects, and duplicates are labeled in a flattened connected component.
17. A system for implementing a self-optimized video analytics pipeline, the system comprising:
a memory; and
one or more processors in communication with the memory configured to:
decode video files into a sequence of frames;
extract features of objects from one or more frames of the sequence of frames of the video files;
employ an adaptive resource allocation component based on reinforcement learning (RL) to dynamically balance resource usage of different microservices included in the video analytics pipeline;
employ an adaptive microservice parameter tuning, component to balance accuracy and performance of a microservice of the different microservices;
apply a graph-based filter to minimize redundant computations across the one or more frames of the sequence of frames; and
apply a deep-learning-based filter to remove unnecessary computations resulting from mismatches between the different microservices in the video analytics pipeline.
18. The system of claim 17 , wherein the graph-based filter removes repeated objects from consecutive frames within the sequence of frames by creating a graph G=(V, E), where each object forms a vertex v i and an edge e (v i , v j ) between two vertices v i and v j , if and only if, these objects arc close to each other, the closeness of two objects determined by spatial closeness and temporal closeness used to compute a distance metric.
19. The system of claim 18 , wherein the spatial closeness employs an overlap ratio between bounding boxes of the objects from the consecutive frames, the overlap ratio given as:
overlap
(
v
i
,
v
j
)
=
area
(
v
i
⋂
v
j
)
min
(
area
(
v
i
)
,
area
(
v
j
)
)
.
20. The system of claim 18 , wherein the temporal closeness employs a lag between the objects from the consecutive frames, the lag given as:
lag
(
v
i
,
v
j
)
=
frame
#
v
i
-
frame
#
v
j
frameInBatch
.Cited by (0)
No later patents cite this yet.
References (0)
No backward citations on record.