Video stabilization using content-aware camera motion estimation

Abstract

Video stabilization is described using content-aware camera motion estimation. In some versions a luminance target frame and a luminance source frame of a sequence of video frames of a scene are received. Motion is extracted from the received luminance target and source frames and the motion is represented as a motion vector field and weights. The weights are divided into a first set of zeros weights for motion in the motion vector field that is near zero motion and a second set of peak weights for motion in the motion field that is not near zero. The zeros weights are compared to a threshold to determine whether there is motion in the scene and if the zeros weights exceed the threshold then selecting a zero motion motion model. A frame of the video sequence is adjusted corresponding to the target frame based on the selected motion model.

What is claimed is: 
     
       1. A method comprising:
 receiving a luminance target frame and a luminance source frame of a sequence of video frames of a scene; 
 extracting motion from the received luminance target and source frames and representing the motion as a motion vector field and weights; 
 dividing the weights into a first set of zeros weights for motion in the motion vector field that is near zero motion and a second set of peak weights for motion in the motion field that is not near zero; 
 comparing the zeros weights to a threshold to determine whether there is motion in the scene and if the zeros weights exceed the threshold then selecting a zero motion motion model; and 
 adjusting a frame of the video sequence corresponding to the target frame based on the selected motion model. 
 
     
     
       2. The method of  claim 1 , further comprising normalizing the luminance of the target frame to the luminance of the source frame. 
     
     
       3. The method of  claim 1 , further comprising determining whether the motion vector field is unreliable and, if the motion vector field is unreliable, then selecting an identity matrix motion model. 
     
     
       4. The method of  claim 1 , further comprising comparing the peak weights to a threshold and, if the peak weights exceed the threshold, then selecting a peak motion motion model. 
     
     
       5. The method of  claim 1 , wherein the luminance target frame is for a long exposure image and the luminance source frame is for a short exposure image. 
     
     
       6. The method of  claim 1 , further comprising modifying the weights before dividing the weights by analyzing the content of the target frame. 
     
     
       7. The method of  claim 1 , further comprising modifying the weights before dividing the weights by using a previous mismatch regions map. 
     
     
       8. The method of  claim 7 , further comprising updating the previous mismatch regions map using spatial and temporal scene analysis. 
     
     
       9. The method of  claim 8 , wherein updating comprises suppressing short term local changes in the scene and suppressing long term changes in the scene. 
     
     
       10. The method of  claim 8 , wherein applying previous mismatch region maps comprises comparing values in the weights to values in a fast previous mismatch region map and in a slow previous mismatch region map and using the least of the determined weight and the weights from the maps as the final weight. 
     
     
       11. The method of  claim 1 , wherein extracting motion comprises applying previous mismatch region maps to suppress regions with short term and long term irregularities. 
     
     
       12. The method of  claim 1 , further comprising modifying the weights before dividing the weights by eliminating flat regions by applying a soft threshold against a variance map on an input region. 
     
     
       13. A computer-readable medium having instructions that when operated on by the computer cause the computer to perform operations comprising:
 receiving a luminance target frame and a luminance source frame of a sequence of video frames of a scene; 
 extracting motion from the received luminance target and source frames and representing the motion as a motion vector field and weights; 
 dividing the weights into a first set of zeros weights for motion in the motion vector field that is near zero motion and a second set of peak weights for motion in the motion field that is not near zero; 
 comparing the zeros weights to a threshold to determine whether there is motion in the scene and if the zeros weights exceed the threshold then selecting a zero motion motion model; and 
 adjusting a frame of the video sequence corresponding to the target frame based on the selected motion model. 
 
     
     
       14. The medium of  claim 13 , the operations further comprising normalizing the luminance of the target frame to the luminance of the source frame. 
     
     
       15. The medium of  claim 13 , the operations further comprising determining whether the motion vector field is unreliable and, if the motion vector field is unreliable, then selecting an identity matrix motion model. 
     
     
       16. A motion video stabilization system comprising:
 an image sensor to record a sequence of video frames; 
 a memory to store the sequence of video frames; 
 a motion extractor engine to extract motion from a luminance target frame and a luminance source frame from the image sensor and to represent the motion as a motion vector field and weights; 
 dividing the weights into a first set of zeros weights for motion in the motion vector field that is near zero motion and a second set of peak weights for motion in the motion field that is not near zero and to compare the zeros weights to a threshold to determine whether there is motion in the scene and if the zeros weights exceed the threshold then selecting a zero motion motion model; and 
 a motion estimation module to adjust a frame of the video sequence corresponding to the target frame based on the selected motion model. 
 
     
     
       17. The system of  claim 16 , further comprising a weight update module to modify the weights before dividing the weights by analyzing the content of the target frame. 
     
     
       18. The system of  claim 16 , further comprising a weight update module to modify the weights before dividing the weights by using a previous mismatch regions map. 
     
     
       19. The system of  claim 18 , further comprising previous mismatch region update module to update the previous mismatch regions map using spatial and temporal scene analysis. 
     
     
       20. The system of  claim 16 , wherein extracting motion comprises applying previous mismatch region maps to suppress regions with short term and long term irregularities.