Methods and systems for managing resolution changes during image processing

Abstract

Systems are configured for performing super-resolution processing for applications by a plurality of different hardware processing units that include hardware processing units that are native to the applications and hardware processing units that are non-native to the applications. Interfaces in the system generate different sets of instructions that are submitted to the different processing units with synchronization objects that synchronize the execution of the instructions.

What is claimed: 
     
         1 . A method for managing resolution changes during image processing, the method comprising:
 identifying rendering capabilities of an image rendering platform, including a capability of the image rendering platform to render images at a first resolution that is greater than or equal to a second resolution of images that a first GPU (graphics processing unit) is configured to generate for an application;   identifying instructions from the application for the first GPU to generate images at the second resolution;   causing the first GPU to generate images for the application at a third resolution that is lower than the second resolution;   submitting the images generated from the first GPU at the third resolution to a secondary processing unit for upscaling the images from the third resolution to the second resolution or the first resolution, where the secondary processing unit is selected from a plurality of processing units, each of the plurality of processing units being configured with one or more super-resolution processing models, the selecting of the secondary processing unit being based on a determination that the secondary processing unit is configured to perform upscaling on image inputs having the third resolution while another processing unit in the plurality of processing units is not configured to perform upscaling on the image input having the third resolution; and   causing the images that have been upscaled by the secondary processing unit to be rendered at the first resolution or the second resolution at the image rendering platform.   
     
     
         2 . The method of  claim 1 , wherein the secondary processing unit comprises a GPU (graphics processing unit) that is non-native to the application. 
     
     
         3 . The method of  claim 1 , wherein the secondary processing unit comprises an NPU (neural processing unit) configured for generating super-resolution images by upscaling images from the third resolution to the first resolution. 
     
     
         4 . The method of  claim 1 , wherein the secondary processing unit comprises an NPU (neural processing unit) configured for generating super-resolution images by upscaling images from the third resolution to the second resolution. 
     
     
         5 . The method of  claim 1 , wherein the method further comprises:
 causing the first GPU to generate images for the application at the third resolution in response to detecting a load on the first GPU that exceeds a predetermined load threshold.   
     
     
         6 . The method of  claim 1 , wherein the method further comprises:
 causing the first GPU to generate images for the application at the third resolution in response to a user-directed instruction to offload processing from the first GPU.   
     
     
         7 . The method of  claim 1 , wherein the method further comprises:
 applying additional image processing effects to the images generated by the first GPU at the third resolution prior to submitting the images to the secondary processing unit for upscaling the images from the third resolution.   
     
     
         8 . The method of  claim 7 , wherein the additional image processing effects comprise auto high dynamic range (HDR) processing effects that alter a color range or a brightness of the image pixels by modifying a luminance of image pixels. 
     
     
         9 . The method of  claim 1 , wherein the method further comprises:
 causing the images that have been upscaled by the secondary processing unit to be further processed by the first GPU prior to final rendering.   
     
     
         10 . The method of  claim 1 , wherein the first GPU and the secondary processing unit are both (i) integrated into a single hardware device, (ii) integrally connected to a shared central processing unit (CPU), and (iii) controlled by a single operating system (OS). 
     
     
         11 . The method of  claim 1 , wherein the secondary processing unit upscales the images from the third resolution to the second resolution, the first resolution being equal to the second resolution. 
     
     
         12 . The method of  claim 1 , wherein the secondary processing unit upscales the images from the third resolution to the first resolution, the first resolution being greater than the second resolution. 
     
     
         13 . A method for managing resolution changes of application images during image processing, the method comprising:
 identifying a selected display resolution that has been selected for rendering application images at a display device;   selecting a super-resolution model for upscaling the application images, the super-resolution model having a target model input image resolution that is different from the selected display resolution and a target model output image resolution that is higher than the target model input image resolution;   subsequent to the selecting of the super-resolution model, causing a graphics processing unit (GPU) to generate the application images with a GPU image output resolution that equals the target model input image resolution; and   applying the super-resolution model to the application images generated by the GPU to generate upscaled application images at the target model output image resolution.   
     
     
         14 . The method of  claim 13 , wherein the GPU is capable of generating images at resolutions that are higher than the GPU image output resolution. 
     
     
         15 . The method of  claim 13 , wherein the GPU image output resolution is lower than the selected display resolution. 
     
     
         16 . The method of  claim 13 , wherein the selected display resolution is selected by an application associated with the application images. 
     
     
         17 . The method of  claim 13 , wherein the selected display resolution is automatically selected based on detected display capabilities of the display device. 
     
     
         18 . The method of  claim 13 , wherein the method further comprises:
 selecting the super-resolution model from a plurality of different super-resolution models that each have a different target model output image resolution and based on the super-resolution model having a target model output image resolution that matches the selected display resolution.   
     
     
         19 . A method for managing resolution changes during image processing by a system that includes a graphics processing unit (GPU), the method comprising:
 selecting, based on display capabilities of a display device, a display resolution for images from an application, the display resolution being greater than or equal to a maximum output resolution of output images that the GPU is configured to generate for the application;   identifying instructions from the application for the GPU to generate images;   causing the GPU to generate the images at an output resolution that is different than the display resolution;   submitting the images generated from the GPU to a secondary processing unit for upscaling the images to the display resolution, where the secondary processing unit is selected from a plurality of processing units, each of the plurality of processing units being configured with one or more super-resolution processing models, the selecting of the secondary processing unit being based on a determination that the secondary processing unit is configured to perform upscaling on image inputs having the output resolution; and   causing the images that have been upscaled by the secondary processing unit to be rendered at the display resolution.