Generating synthetic video frames using optical flow

Abstract

A novel method of using optical flow algorithm that maximizes the benefit of optical flow synthetic frames while minimizing the associated computation cost is provided. When using optical flow to produce synthetic frames between two actual/recorded frames, the method computes a set of estimates of optical flows (or a flow estimate) between the two frames. These flow estimates are then used to compute all synthetic frames that are needed between the two actual frames by interpolation, which creates each synthetic frame based on its temporal distances from the pair of actual frames.

What is claimed is: 
     
         1 . A method for generating synthetic frames, comprising:
 in a computer processor with a memory and a data storage,
 receiving two actual image frames; 
 creating a luma level histogram for each of the actual frames; 
 determining a sum of each luma level histogram for each of the actual frames; 
 comparing the determined sum for each actual frame to a threshold; 
 if the compared determined sums are below the threshold,
 computing a set of estimated optical flows between the two actual frames by,
 computing a temporal distance between the two actual frames; 
 generating, by interpolation, synthetic frames needed between the two actual frames. 
 
 
   
     
     
         2 . The method of  claim 1  wherein the number of estimated optical flows in the set is a predetermined number. 
     
     
         3 . The method of  claim 2  wherein the predetermined number is based on the computer processor and memory capabilities. 
     
     
         4 . The method of  claim 1  further comprising, detecting distortions in at least one of the two actual frames. 
     
     
         5 . The method of  claim 1  wherein the luma level is between 0 and 255. 
     
     
         6 . The method of  claim 1  wherein if the compared determined sums are above the threshold,
 computing interpolated synthetic frames between the actual frames using non-optical flow based frame interpolation. 
 
     
     
         7 . The method of  claim 1  wherein the comparing of determined sums includes computing the difference between the luma light levels. 
     
     
         8 . A non-transitory computer-readable medium having computer-executable instructions thereon for a method of generating synthetic frames, the method comprising:
 receiving two actual image frames;   creating a luma level histogram for each of the actual frames;   determining a sum of each luma level histogram for each of the actual frames;   comparing the determined sum for each actual frame to a threshold;   if the compared determined sums are below the threshold,
 computing a set of estimated optical flows between the two actual frames by,
 computing a temporal distance between the two actual frames; 
 generating, by interpolation, synthetic frames needed between the two actual frames. 
 
   
     
     
         9 . The method of  claim 8  wherein the number of estimated optical flows in the set is a predetermined number. 
     
     
         10 . The method of  claim 9  wherein the predetermined number is based on the computer processor and memory capabilities. 
     
     
         11 . The method of  claim 8  further comprising, detecting distortions in at least one of the two actual frames. 
     
     
         12 . The method of  claim 8  wherein the luma level is between 0 and 255. 
     
     
         13 . The method of  claim 8  wherein if the compared determined sums are above the threshold,
 computing interpolated synthetic frames between the actual frames using non-optical flow based frame interpolation. 
 
     
     
         14 . The method of  claim 8  wherein the comparing of determined sums includes computing the difference between the luma light levels. 
     
     
         15 . A system for generating synthetic frames, comprising:
 a computer processor with a memory and a data storage, the computer configured to,
 receive two actual image frames; 
 create a luma level histogram for each of the actual frames; 
 determine a sum of each luma level histogram for each of the actual frames; 
 compare the determined sum for each actual frame to a threshold; 
 if the compared determined sums are below the threshold,
 compute a set of estimated optical flows between the two actual frames by,
 compute a temporal distance between the two actual frames; 
 generate, by interpolation, synthetic frames needed between the two actual frames. 
 
 
   
     
     
         16 . The system of  claim 15  wherein the number of estimated optical flows in the set is a predetermined number. 
     
     
         17 . The system of  claim 16  wherein the predetermined number is based on the computer processor and memory capabilities. 
     
     
         18 . The system of  claim 15  wherein the computer is further configured to, detect distortions in at least one of the two actual frames. 
     
     
         19 . The system of  claim 15  wherein the luma level is between 0 and 255.