Image upscaling

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for upscaling an image. One of the methods includes upscaling a low resolution image, creating first pixel subsets of the first upscaled image, creating second pixel subsets of a high resolution image, determining, for each subset in the pixel subsets, a value of a property of the pixel subset, determining, for each subset in the pixel subsets, a group of subsets to which the corresponding pixel subset belongs using the value of the property, and determining, for each of the groups of subsets, a filter to apply to each of the first pixel subsets that correspond to the pixel subsets in the group to create a final pixel subset that approximates the corresponding second pixel subset using the first pixel subset, a combination of all of the final pixel subsets representing a second upscaled image.

What is claimed is: 
     
       1. A system comprising:
 a data processing apparatus; and 
 a non-transitory computer readable storage medium in data communication with the data processing apparatus and storing instructions executable by the data processing apparatus and upon such execution cause the data processing apparatus to perform operations comprising:
 upscaling a low resolution image with a first resolution that depicts particular content to a second resolution greater than the first resolution to create a first upscaled image; 
 creating first pixel subsets of the first upscaled image, each subset in the first pixel subsets having a first size, each pixel in the first upscaled image included in a subset in the first pixel subsets; 
 creating second pixel subsets of a high resolution image with a third resolution, greater than the first resolution, that depicts the particular content, each subset in the second pixel subsets having a second size, each pixel in the high resolution image included in a subset in the second pixel subsets, a quantity of the first pixel subsets being the same as a quantity of the second pixel subsets and each of the first pixel subsets corresponding to a respective one of the second pixel subsets, wherein the high resolution image is a separate image from the first upscaled image; 
 determining, for each subset in the first pixel subsets or for each subset in the second pixel subsets, a value of a property of the corresponding pixel subset using a) the corresponding pixel subset and b) at least one pixel subset adjacent to the corresponding pixel subset that includes one or more pixels that are not included in the corresponding pixel subset; 
 determining, for each subset in the first pixel subsets or for each subset in the second pixel subsets, a group of subsets to which the corresponding pixel subset belongs using the value of the property of the corresponding pixel subset, each pixel subset included in one and only one group; and 
 determining, for each of the groups of subsets, a filter to apply to each of the first pixel subsets that correspond to the pixel subsets in the group to create a final pixel subset that approximates the corresponding second pixel subset using the first pixel subset, a combination of all of the final pixel subsets representing a second upscaled image. 
 
 
     
     
       2. The system of  claim 1 , the operations comprising:
 receiving the low resolution image prior to upscaling the low resolution image; and 
 receiving the high resolution image prior to creating the second pixel subsets of the high resolution image. 
 
     
     
       3. The system of  claim 1 , wherein upscaling the low resolution image with the first resolution that depicts particular content to the second resolution greater than the first resolution to create the first upscaled image comprises upscaling the low resolution image to the third resolution of the high resolution image. 
     
     
       4. The system of  claim 1 , wherein:
 the first size is the same size as the second size; and 
 creating the second pixel subsets of the high resolution image, each subset in the second pixel subsets having the second size comprises creating the second pixel subsets of the high resolution image, each subset in the second pixel subsets having the first size. 
 
     
     
       5. The system of  claim 1 , wherein:
 creating the first pixel subsets of the first upscaled image comprises creating first non-overlapping pixel subsets of the first upscaled image; and 
 creating second pixel subsets of the high resolution image with the third resolution comprises creating second non-overlapping pixel subsets of the high resolution image. 
 
     
     
       6. The system of  claim 1 , wherein:
 creating the first pixel subsets of the first upscaled image comprises creating first overlapping pixel subsets of the first upscaled image; and 
 creating second pixel subsets of the high resolution image with the third resolution comprises creating second overlapping pixel subsets of the high resolution image. 
 
     
     
       7. The system of  claim 1 , wherein determining, for each subset in the first pixel subsets or for each subset in the second pixel subsets, the value of the property of the corresponding pixel subset comprises determining, for each subset in the first pixel subsets or for each subset in the second pixel subsets, an angle of an edge in the corresponding pixel subset. 
     
     
       8. The system of  claim 1 , wherein determining, for each subset in the first pixel subsets or for each subset in the second pixel subsets, the value of the property of the corresponding pixel subset comprises determining, for each subset in the first pixel subsets or for each subset in the second pixel subsets, a derivative of an edge in the corresponding pixel subset. 
     
     
       9. The system of  claim 1 , wherein determining, for each subset in the first pixel subsets or for each subset in the second pixel subsets, the value of the property of the corresponding pixel subset comprises determining, for each subset in the first pixel subsets or for each subset in the second pixel subsets, the value of the property of the corresponding pixel subset using the corresponding first pixel subset and all pixel subsets adjacent to the corresponding pixel subset that each include one or more pixels that are not included in the corresponding pixel subset. 
     
     
       10. The system of  claim 1 , wherein:
 determining, for each of the groups of subsets, the filter comprises determining, during a training process and for each of the groups of subsets, the filter; and 
 the operations comprise compressing, during the training process, the low resolution image prior to upscaling the low resolution image. 
 
     
     
       11. The system of  claim 1 , wherein:
 determining, for each of the groups of subsets, the filter comprises determining, during a training process and for each of the groups of subsets, the filter; and 
 the operations comprise sharpening, during the training process, the high resolution image prior to creating the second pixel subsets of the high resolution image. 
 
     
     
       12. The system of  claim 1 , the operations comprising:
 determining, for each of the groups of subsets, a second filter to apply to each of the final pixel subsets in the group to create a second final pixel subset that approximates the corresponding second pixel subset using the final pixel subset, a combination of all of the second final pixel subsets representing a third upscaled image. 
 
     
     
       13. The system of  claim 1 , wherein determining, for each of the groups of subsets, the filter to apply to each of the first pixel subsets that correspond to the pixel subsets in the group to create the final pixel subset that approximates the corresponding second pixel subset using the first pixel subset comprises:
 determining, for each of the groups of subsets, a horizontal filter to apply to each of the first pixel subsets that correspond to the pixel subsets in the group; and 
 determining, for each of the groups of subsets, a vertical filter to apply to each of the first pixel subsets that correspond to the pixel subsets in the group. 
 
     
     
       14. The system of  claim 1 , wherein determining, for each of the groups of subsets, the filter to apply to each of the first pixel subsets that correspond to the pixel subsets in the group to create the final pixel subset that approximates the corresponding second pixel subset using the first pixel subset comprises determining, for each of the groups of subsets, a unique filter to apply to each of the first pixel subsets that correspond to the pixel subsets in the group that is different from the filters for the other groups. 
     
     
       15. The system of  claim 1 , wherein the data processing apparatus automatically upscales the low resolution image to create the first upscaled image. 
     
     
       16. The system of  claim 1 , comprising a mobile device that includes the data processing apparatus and the computer readable storage medium. 
     
     
       17. The system of  claim 1 , comprising a personal computer that includes the data processing apparatus and the computer readable storage medium. 
     
     
       18. A system comprising:
 a processor to upscale a low resolution image with a first resolution that depicts particular content to a second resolution greater than the first resolution to create a first upscaled image; 
 a data processing apparatus; and 
 a non-transitory computer readable storage medium in data communication with the data processing apparatus and storing instructions executable by the data processing apparatus and upon such execution cause the data processing apparatus to perform operations comprising:
 creating first pixel subsets of the first upscaled image, each subset in the first pixel subsets having a first size, each pixel in the first upscaled image included in a subset in the first pixel subsets; 
 creating second pixel subsets of a high resolution image with a third resolution, greater than the first resolution, that depicts the particular content, each subset in the second pixel subsets having a second size, each pixel in the high resolution image included in a subset in the second pixel subsets, a quantity of the first pixel subsets being the same as a quantity of the second pixel subsets and each of the first pixel subsets corresponding to a respective one of the second pixel subsets, wherein the high resolution image is a separate image from the first upscaled image; 
 determining, for each subset in the first pixel subsets or for each subset in the second pixel subsets, a value of a property of the corresponding pixel subset using a) the corresponding pixel subset and b) at least one pixel subset adjacent to the corresponding pixel subset that includes one or more pixels that are not included in the corresponding pixel subset; 
 determining, for each subset in the first pixel subsets or for each subset in the second pixel subsets, a group of subsets to which the corresponding pixel subset belongs using the value of the property of the corresponding pixel subset, each pixel subset included in one and only one group; and 
 determining, for each of the groups of subsets, a filter to apply to each of the first pixel subsets that correspond to the pixel subsets in the group to create a final pixel subset that approximates the corresponding second pixel subset using the first pixel subset, a combination of all of the final pixel subsets representing a second upscaled image. 
 
 
     
     
       19. The system of  claim 18 , wherein the processor comprises an image and signal processor that is different from the data processing apparatus. 
     
     
       20. A non-transitory computer readable storage medium storing instructions executable by a data processing apparatus and upon such execution cause the data processing apparatus to perform operations comprising:
 creating first pixel subsets of a low resolution image with a first resolution that depicts particular content, each subset in the first pixel subsets having a first size, each pixel in the low resolution image included in a subset in the first pixel subsets; 
 creating second pixel subsets of a high resolution image with a second resolution, greater than the first resolution, that depicts the particular content, each subset in the second pixel subsets having a second size, each pixel in the high resolution image included in a subset in the second pixel subsets, a quantity of the first pixel subsets being the same as a quantity of the second pixel subsets and each of the first pixel subsets corresponding to a respective one of the second pixel subsets; 
 determining, for each subset in the first pixel subsets or for each subset in the second pixel subsets, a value of a property of the corresponding pixel subset using a) the corresponding pixel subset and b) at least one pixel subset adjacent to the corresponding pixel subset that includes one or more pixels that are not included in the corresponding pixel subset; 
 determining, for each subset in the first pixel subsets or for each subset in the second pixel subsets, a group of subsets to which the corresponding pixel subset belongs using the value of the property of the corresponding pixel subset, each pixel subset included in one and only one group; and 
 determining, for each of the groups of subsets, a filter to apply to each of the first pixel subsets that correspond to the pixel subsets in the group to create a final pixel subset that approximates the corresponding second pixel subset using the first pixel subset, a combination of all of the final pixel subsets representing an upscaled image that depicts the particular content. 
 
     
     
       21. The computer readable storage medium of  claim 20 , wherein determining, for each of the groups of subsets, the filter to apply to each of the first pixel subsets that correspond to the pixel subsets in the group to create the final pixel subset that approximates the corresponding second pixel subset using the first pixel subset comprises creating a single filter that upscales the pixel subset and creates the final pixel subset that approximates the corresponding second pixel subset using the first pixel subset. 
     
     
       22. A computer-implemented method comprising:
 upscaling a low resolution image with a first resolution that depicts particular content to a second resolution greater than the first resolution to create a first upscaled image; 
 creating first pixel subsets of the first upscaled image, each subset in the first pixel subsets having a first size, each pixel in the first upscaled image included in a subset in the first pixel subsets; 
 creating second pixel subsets of a high resolution image with a third resolution, greater than the first resolution, that depicts the particular content, each subset in the second pixel subsets having a second size, each pixel in the high resolution image included in a subset in the second pixel subsets, a quantity of the first pixel subsets being the same as a quantity of the second pixel subsets and each of the first pixel subsets corresponding to a respective one of the second pixel subsets, wherein the high resolution image is a different image than the first upscaled image, wherein the high resolution image is a separate image from the first upscaled image; 
 determining, for each subset in the first pixel subsets or for each subset in the second pixel subsets, a value of a property of the corresponding pixel subset using a) the corresponding pixel subset and b) at least one pixel subset adjacent to the corresponding pixel subset that includes one or more pixels that are not included in the corresponding pixel subset; 
 determining, for each subset in the first pixel subsets or for each subset in the second pixel subsets, a group of subsets to which the corresponding pixel subset belongs using the value of the property of the corresponding pixel subset, each pixel subset included in one and only one group; and 
 determining, for each of the groups of subsets, a filter to apply to each of the first pixel subsets that correspond to the pixel subsets in the group to create a final pixel subset that approximates the corresponding second pixel subset using the first pixel subset, a combination of all of the final pixel subsets representing a second upscaled image, wherein a first filter for a first group in the groups of subsets is a different filter than a second filter for a second group in the groups of subsets.