Scaling factor detection for compressed images and videos

Abstract

Detection of scaling of compressed videos or images is provided. A frequency domain transformation is applied along both horizontal and vertical directions of input video or images to generate frequency domain data. Statistics in the frequency domain data are computed for each of the horizontal and vertical directions to extract features. The features are modeled to scores along each of the horizontal and vertical directions. An original resolution of the input video or images in the horizontal and vertical directions is identified according to the scores.

What is claimed is: 
     
       1. A method for detecting scaling of compressed videos or images, comprising:
 applying a frequency domain transformation along both horizontal and vertical directions of input video or images to generate frequency domain data, the input video or images having been scaled up from an original resolution to a scaled resolution greater than the original resolution; 
 computing statistics in the frequency domain data for each of the horizontal and vertical directions to extract features; 
 modeling the features to scores along each of the horizontal and vertical directions; and 
 identifying the original resolution of the input video or images in the horizontal and vertical directions according to the scores. 
 
     
     
       2. The method of  claim 1 , wherein the frequency domain transformation includes one or more of 1D or 2D Discrete Cosine Transformation (DCT) or Fast Fourier Transform (FFT). 
     
     
       3. The method of  claim 2 , wherein the frequency domain data includes DCT coefficients formed as a DCT spectrum computed for each of the horizontal and vertical directions, and the statistics in the frequency domain data include one or more of:
 (i) a horizontal mean of absolute DCT coefficients of the DCT spectrum and a vertical mean of absolute DCT coefficients of the DCT spectrum, 
 (ii) first and second derivatives of the horizontal and vertical means of the absolute DCT coefficients of the DCT spectrum, and/or 
 (iii) different order statistics of the horizontal and vertical means of the absolute DCT coefficients of the DCT spectrum. 
 
     
     
       4. The method of  claim 1 , further comprising:
 applying the frequency domain transformation to a sub-frame including a subset of rows or columns of the input video or images and calculating a corresponding sub-sampling ratio; 
 identifying the original resolution for the subset of rows or columns of the input video or images; and 
 normalizing the detected sub-frame resolution by the sub-sampling ratio to determine the original resolution. 
 
     
     
       5. The method of  claim 1 , further comprising:
 applying the frequency domain transformation to a plurality of subsets of rows or columns of the input video or images and calculating corresponding sub-sampling ratios for each of the plurality of subsets of rows or columns; 
 detecting, according to the sub-sampling ratios, scaling factors for identifying the original resolution for each of the plurality of subsets of rows or columns of the input video or images; 
 normalizing the detected resolution for each of the plurality of subsets of rows or columns by the corresponding sub-sampling ratio for the respective subset of rows or columns to determine the original resolution; and 
 weighting the original resolution for each of the plurality of subsets of rows or columns to determine the original resolution of the input video or images as a whole. 
 
     
     
       6. The method of  claim 5 , wherein the weighting includes one or more of applying a pooling strategy including one or more of direct averaging or weighted averaging to determine the original resolution of the input video or images as a whole, wherein the weighted averaging includes one or more of: distortion/quality based weighting, entropy/information based weighting, or saliency/visual attention based weighting. 
     
     
       7. The method of  claim 1 , further comprising:
 computing statistics in the frequency domain data spectrum for each of the horizontal and vertical directions, the statistics including one or more of: 
 (i) a horizontal mean of absolute DCT coefficients of the DCT spectrum and a vertical mean of absolute DCT coefficients of the DCT spectrum, 
 (ii) first and second derivatives of the horizontal and vertical means of the absolute DCT coefficients of the DCT spectrum, and/or 
 (iii) different order statistics of the horizontal and vertical means of the absolute DCT coefficients of the DCT spectrum; and 
 identifying an overall score indicative of the original resolution of the input video or images using a reward scoring function applying one or more rewards or penalties to the statistics. 
 
     
     
       8. The method of  claim 7 , further comprising using one or more checking list procedures to improve accuracy in determination of the original resolution, the checking list procedures including one or more of: (i) checking common widths, heights, and their combinations and assign them different rewards, such that more common resolutions are selected for; (ii) checking and penalizing an aspect ratio change, as a change in aspect ratio may be less likely than a scaling maintaining the aspect ratio; (iii) giving small tolerance when predictions are very close to display resolution; and/or (iv) abandoning both dimensions when one of the dimensions is the same as the display resolution. 
     
     
       9. The method of  claim 8 , further comprising:
 categorizing the input video or images into one category of a plurality of categories; and 
 varying one or more of the reward scoring function or the checking list procedures according to the one category. 
 
     
     
       10. The method of  claim 9 , wherein the plurality of categories includes a set of codec types, a set of display resolutions, or a set of aspect ratios. 
     
     
       11. The method of  claim 8 , further comprising:
 decomposing the input video or images into a plurality of decompositions; and 
 for each of the plurality of decompositions
 using an overall scoring function corresponding to the respective decomposition, and 
 using a final checking list corresponding to the respective decomposition. 
 
 
     
     
       12. The method of  claim 11 , wherein the plurality of decompositions includes one or more of: a plurality of different groups of frames, a plurality of different content types, a plurality of different distortion types, a plurality of different complexity levels, or a plurality of different quality levels. 
     
     
       13. A system for detecting scaling of compressed videos or images, comprising:
 a computing device programmed to
 apply a frequency domain transformation along both horizontal and vertical directions of input video or images to generate frequency domain data, the input video or images having been scaled up from an original resolution to a scaled resolution greater than the original resolution; 
 compute statistics in the frequency domain data for each of the horizontal and vertical directions to extract features; 
 model the features to scores along each of the horizontal and vertical directions; and 
 identify the original resolution of the input video or images in the horizontal and vertical directions according to the scores. 
 
 
     
     
       14. The system of  claim 13 , wherein the frequency domain transformation includes one or more of 1D or 2D Discrete Cosine Transformation (DCT) or Fast Fourier Transform (FFT). 
     
     
       15. The system of  claim 14 , wherein the frequency domain data includes DCT coefficients formed as a DCT spectrum computed for each of the horizontal and vertical directions, and the statistics in the frequency domain data include one or more of:
 (i) a horizontal mean of absolute DCT coefficients of the DCT spectrum and a vertical mean of absolute DCT coefficients of the DCT spectrum, 
 (ii) first and second derivatives of the horizontal and vertical means of the absolute DCT coefficients of the DCT spectrum, 
 (iii) and/or different order statistics of the horizontal and vertical means of the absolute DCT coefficients of the DCT spectrum. 
 
     
     
       16. The system of  claim 13 , wherein the computing device is further programmed to:
 applying the frequency domain transformation to a sub-frame including a subset of rows or columns of the input video or images and calculating a corresponding sub-sampling ratio; 
 identify the original resolution for the subset of rows or columns of the input video or images; and 
 normalize the detected sub-frame resolution by the sub-sampling ratio to determine the original resolution. 
 
     
     
       17. The system of  claim 13 , wherein the computing device is further programmed to:
 apply the frequency domain transformation to a plurality of subsets of rows or columns of the input video or images and calculating corresponding sub-sampling ratios for each of the plurality of subsets of rows or columns; 
 detect, according to the sub-sampling ratios, scaling factors for identifying the original resolution for each of the plurality of subsets of rows or columns of the input video or images; 
 normalize the detected resolution for each of the plurality of subsets of rows or columns by the corresponding sub-sampling ratio for the respective subset of rows or columns to determine the original resolution; and 
 weight the original resolution for each of the plurality of subsets of rows or columns to determine the original resolution of the input video or images as a whole. 
 
     
     
       18. The system of  claim 17 , wherein the weighting includes one or more of applying a pooling strategy including one or more of direct averaging or weighted averaging to determine the original resolution of the input video or images as a whole, wherein the weighted averaging includes one or more of: distortion/quality based weighting, entropy/information based weighting, or saliency/visual attention based weighting. 
     
     
       19. The system of  claim 13 , wherein the computing device is further programmed to:
 compute statistics in the frequency domain data, the statistics including one or one or more of: 
 (i) a horizontal mean of absolute DCT coefficients of the DCT spectrum and a vertical mean of absolute DCT coefficients of the DCT spectrum, 
 (ii) first and second derivatives of the horizontal and vertical means of the absolute DCT coefficients of the DCT spectrum, and/or 
 (iii) different order statistics of the horizontal and vertical means of the absolute DCT coefficients of the DCT spectrum; and 
 identify an overall score indicative of the original resolution of the input video or images using a reward scoring function applying one or more rewards or penalties to the statistics. 
 
     
     
       20. The system of  claim 19 , wherein the computing device is further programmed to use one or more checking list procedures to improve accuracy in determination of the original resolution, the checking list procedures including one or more of: (i) checking common widths, heights, and their combinations and assign them different rewards, such that more common resolutions are selected for; (ii) checking and penalizing an aspect ratio change, as a change in aspect ratio may be less likely than a scaling maintaining the aspect ratio; (iii) giving small tolerance when predictions are very close to display resolution; and/or (iv) abandoning both dimensions when one of the dimensions is the same as the display resolution. 
     
     
       21. The system of  claim 20 , wherein the computing device is further programmed to:
 categorize the input video or images into one category of a plurality of categories; and 
 vary one or more of the reward scoring function or the checking list procedures according to the one category. 
 
     
     
       22. The system of  claim 21 , wherein the plurality of categories includes a set of codec types, a set of display resolutions, or a set of aspect ratios. 
     
     
       23. The system of  claim 20 , wherein the computing device is further programmed to:
 decompose the input video or images into a plurality of decompositions; and 
 for each of the plurality of decompositions
 use an overall scoring function corresponding to the respective decomposition, and 
 use a final checking list corresponding to the respective decomposition. 
 
 
     
     
       24. The system of  claim 23 , wherein the plurality of decompositions includes one or more of: a plurality of different groups of frames, a plurality of different content types, a plurality of different distortion types, a plurality of different complexity levels, or a plurality of different quality levels.