Adaptive path smoothing for video stabilization

Abstract

Techniques and architectures for video stabilization can transform a shaky video to a steady-looking video. A path smoothing process can generate an optimized camera path for video stabilization. With a large smoothing kernel, a path smoothing process can remove both high frequency jitters and low frequency bounces, and at the same time can preserve discontinuous camera motions (such as quick panning or scene transition) to avoid excessive cropping or geometry distortion. A sliding window based implementation includes a path smoothing process that can be used for real-time video stabilization.

What is claimed is: 
     
       1. A system comprising:
 a video input port to receive a video comprising a sequence of video frames; 
 one or more processors; and 
 memory storing instructions that, when executed by the one or more processors, configure the one or more processors to perform operations comprising:
 identifying object features in individual video frames of the sequence of the video frames; 
 performing homographic estimation between video frames of the sequence of video frames to generate a modified sequence of video frames, wherein the homographic estimation is based, at least in part, on the object features identified; 
 determining estimation errors among video frames of the modified sequence of video frames; and 
 applying adaptive path smoothing to the modified sequence of video frames to generate a smoothed sequence of video frames, wherein the adaptive path smoothing is based, at least in part, on changes among the individual video frames of the sequence of the video frames and on the estimation errors. 
 
 
     
     
       2. The system of  claim 1 , further comprising a video partition module communicatively connected to the video input port, the video partition module configured to perform operations comprising:
 receiving the video from the video input port; 
 partitioning the video into the sequence of video frames and additional sequences of video frames; 
 applying frame delays to individual sequences of the additional sequences of video frames; and 
 respectively providing the frame delayed additional sequences to individual buffers of the one or more processors. 
 
     
     
       3. The system of  claim 2 , wherein the video partition module is stored in the memory and executable by the one or more processors. 
     
     
       4. The system of  claim 2 , wherein applying the frame delays further comprises frame-shifting video frames of the additional sequences so that the additional sequences of video frames are respectively offset from the sequence of video frames by m×n frames, wherein m is the series of integers 1, 2, 3 . . . and n is a predetermined number. 
     
     
       5. The system of  claim 1 , further comprising a video aggregation module configured to perform operations comprising:
 combining the smoothed sequence of video frames with other smoothed sequences of video frames to produce an aggregated video portion; and 
 
       applying post-process filtering to smooth the aggregated video portion. 
     
     
       6. The system of  claim 5 , wherein the video aggregation module is stored in the memory and executable by the one or more processors. 
     
     
       7. The system of  claim 5 , wherein the other smoothed sequences of video frames are based, at least in part, on sequences of frame-shifted video frames that are respectively offset from the sequence of video frames by m×n frames, wherein m is the series of integers 1, 2, 3 . . . and n is a particular number. 
     
     
       8. The system of  claim 1 , wherein the object features in the individual video frames of the sequence of the video frames comprise points or edges in the individual video frames. 
     
     
       9. The system of  claim 1 , wherein the one or more processors comprise one or more central processing units (CPUs), graphics processing units (GPUs), or video buffer processors. 
     
     
       10. Computer-readable storage media storing computer-executable instructions that, when executed by one or more processors, configure the one or more processors to perform operations comprising:
 partitioning a video into a plurality of video sequences; 
 delaying individual video sequences of the plurality of video sequences so that the individual video sequences are delayed by a particular number of frames with respect to one another; and 
 for each video sequence of the individual video sequences, perform a smoothing process comprising:
 identifying object features in individual video frames of the video sequence; 
 generating mappings among the individual video frames based, at least in part, on the object features identified in the individual video frames to generate a modified video sequence; 
 determining estimation errors among video frames of the modified video sequence; and 
 
 applying adaptive path smoothing to the modified video sequence to generate a smoothed video sequence, wherein the adaptive path smoothing is based, at least in part, on changes among the individual video frames of the video sequence and on the estimation errors. 
 
     
     
       11. The computer-readable storage medium of  claim 10 , the operations further comprising:
 combining the smoothed video sequence with other smoothed video sequences that are based, at least in part, on other video sequences of the individual video sequences to produce an aggregated video portion; and 
 applying post-process filtering to smooth the aggregated video portion. 
 
     
     
       12. The computer-readable storage medium of  claim 10 , wherein the object features in the individual video frames of the sequence of the video frames comprise points or edges in the individual video frames. 
     
     
       13. The computer-readable storage medium of  claim 10 , wherein the smoothing process is performed substantially simultaneously for more than one of the individual video sequences. 
     
     
       14. The computer-readable storage medium of  claim 10 , wherein generating mappings among the individual video frames further comprises performing homographic estimations among the individual video frames. 
     
     
       15. A method comprising:
 receiving at a video port a video portion comprising a sequence of video frames; 
 comparing affine object features of individual video frames of the sequence of the video frames to affine object features of other individual video frames of the sequence of the video frames; 
 performing homographic estimation between or among video frames of the sequence of video frames to generate a modified sequence of video frames, wherein the homographic estimation is based, at least in part, on the comparing; 
 determining estimation errors among video frames of the modified sequence of video frames; and 
 applying, by one or more processors, adaptive path smoothing to the modified sequence of video frames to generate a smoothed sequence of video frames, wherein the adaptive path smoothing preserves motion discontinuities corresponding to changes of images among the sequence of video frames, and wherein the adaptive path smoothing is based, at least in part, on the changes and on the estimation errors. 
 
     
     
       16. The method of  claim 15 , further comprising:
 combining the smoothed sequence of video frames with other smoothed sequences of video frames to produce an aggregated video portion; and 
 applying post-process filtering to smooth the aggregated video portion. 
 
     
     
       17. The method of  claim 16 , wherein the other smoothed sequences of video frames are based, at least in part, on sequences of frame-shifted video frames that are respectively offset from the sequence of video frames by m×n frames, wherein m is the series of integers 1, 2, 3 . . . and n is a particular number. 
     
     
       18. The method of  claim 15 , wherein the affine object features in the individual video frames of the sequence of the video frames comprise points or edges in the individual video frames. 
     
     
       19. The method of  claim 15 , wherein at least a portion of the changes correspond to panning, rotation, and/or scene transition in the video portion. 
     
     
       20. The method of  claim 15 , wherein performing the homographic estimation further comprises performing the homographic estimation between two consecutive video frames of the sequence of the video frames.