US10789675B2ActiveUtilityA1

Apparatus and method for correcting image regions following upsampling or frame interpolation

88
Assignee: INTEL CORPPriority: Dec 28, 2018Filed: Dec 28, 2018Granted: Sep 29, 2020
Est. expiryDec 28, 2038(~12.5 yrs left)· nominal 20-yr term from priority
Inventors:Daniel Pohl
G06T 15/06G06T 1/60G06T 1/20G06T 3/4007G06T 7/20G06T 5/50G06T 2207/10016G06T 2207/20081G06T 5/005G06T 5/90G06T 5/77
88
PatentIndex Score
4
Cited by
7
References
27
Claims

Abstract

Apparatus and method for correcting image regions following upsampling or frame interpolation. For example, one embodiment of an apparatus comprises a machine-learning engine to evaluate at least a first image in a sequence of images generated by a real-time interactive application, the machine learning engine to responsively use previously learned data to generate an upsampled or interpolated image comprising a plurality of pixel patches. In one embodiment, each pixel patch is associated with a confidence value reflecting how accurately the pixel patch was generated by the machine learning engine. A selective ray tracing engine identifies a first pixel patch to be corrected based a first confidence value corresponding to the first pixel patch being lower than a threshold and performs ray tracing operations on a first portion of the first image to generate a corrected first pixel patch.

Claims

exact text as granted — not AI-modified
What is claimed is: 
     
       1. An apparatus comprising:
 a machine-learning engine to evaluate at least a first image in a sequence of images generated by a real-time interactive application, the machine learning engine to responsively use previously learned data to generate an upsampled or interpolated image comprising a plurality of pixel patches, 
 wherein each pixel patch is associated with a confidence value reflecting how accurately the pixel patch was generated by the machine learning engine; and 
 a selective ray tracing engine to identify a first pixel patch to be corrected based a first confidence value associated with the first pixel patch being lower than a threshold, the selective ray tracing engine to perform ray tracing operations on a first portion of the first image corresponding to the first pixel patch to generate a corrected first pixel patch. 
 
     
     
       2. The apparatus of  claim 1  wherein the corrected first pixel patch is to replace the first pixel patch in the upsampled or interpolated image prior to displaying the upsampled or interpolated image on a display. 
     
     
       3. The apparatus of  claim 1  wherein the first confidence value comprises a confidence value generated by the machine-learning engine when generating the upsampled or interpolated image. 
     
     
       4. The apparatus of  claim 1  wherein the first confidence value is generated by evaluating motion vectors of graphical objects across the sequence of images. 
     
     
       5. The apparatus of  claim 4  further comprising:
 an image interpolator to generate the first image by interpolating from two or more other images in the sequence of images the image interpolator to use the motion vectors to perform the frame interpolation to generate the first image. 
 
     
     
       6. The apparatus of  claim 1  further comprising:
 a rendering engine to generate the sequence of images including the first image. 
 
     
     
       7. The apparatus of  claim 6  wherein the rendering engine comprises rasterization circuitry and/or logic. 
     
     
       8. The apparatus of  claim 6  wherein the rendering engine comprises ray tracing circuitry and/or logic. 
     
     
       9. The apparatus of  claim 1  wherein the first portion of the first image comprises N×N pixels and the first pixel patch comprises M×M pixels, where N<M. 
     
     
       10. A method comprising:
 evaluating at least a first image in a sequence of images generated by a real-time interactive application; 
 responsively using previously learned data to generate an upsampled or interpolated image comprising a plurality of pixel patches, wherein each pixel patch is associated with a confidence value reflecting how accurately the pixel patch was generated; 
 identifying a first pixel patch to be corrected based a first confidence value associated with the first pixel patch being lower than a threshold; and 
 performing ray tracing operations on a first portion of the first image corresponding to the first pixel patch to generate a corrected first pixel patch. 
 
     
     
       11. The method of  claim 10  wherein the corrected first pixel patch is to replace the first pixel patch in the upsampled or interpolated image prior to displaying the upsampled or interpolated image on a display. 
     
     
       12. The method of  claim 10  wherein the first confidence value comprises a confidence value generated by the machine-learning engine when generating the upsampled or interpolated image. 
     
     
       13. The method of  claim 10  wherein the first confidence value is generated by evaluating motion vectors of graphical objects across the sequence of images. 
     
     
       14. The method of  claim 13  further comprising:
 an image interpolator to generate the first image by interpolating from two or more other images in the sequence of images the image interpolator to use the motion vectors to perform the frame interpolation to generate the first image. 
 
     
     
       15. The method of  claim 10  further comprising:
 a rendering engine to generate the sequence of images including the first image. 
 
     
     
       16. The method of  claim 15  wherein the rendering engine comprises rasterization circuitry and/or logic. 
     
     
       17. The method of  claim 15  wherein the rendering engine comprises ray tracing circuitry and/or logic. 
     
     
       18. The method of  claim 10  wherein the first portion of the first image comprises N×N pixels and the first pixel patch comprises M×M pixels, where N<M. 
     
     
       19. A non-transitory machine-readable medium having program code stored thereon which, when executed by a machine, causes the machine to perform the operations of:
 evaluating at least a first image in a sequence of images generated by a real-time interactive application; 
 responsively using previously learned data to generate an upsampled or interpolated image comprising a plurality of pixel patches, wherein each pixel patch is associated with a confidence value reflecting how accurately the pixel patch was generated; 
 identifying a first pixel patch to be corrected based a first confidence value associated with the first pixel patch being lower than a threshold; and 
 performing ray tracing operations on a first portion of the first image corresponding to the first pixel patch to generate a corrected first pixel patch. 
 
     
     
       20. The non-transitory machine-readable medium of  claim 19  wherein the corrected first pixel patch is to replace the first pixel patch in the upsampled or interpolated image prior to displaying the upsampled or interpolated image on a display. 
     
     
       21. The non-transitory machine-readable medium of  claim 19  wherein the first confidence value comprises a confidence value generated by the machine-learning engine when generating the upsampled or interpolated image. 
     
     
       22. The non-transitory machine-readable medium of  claim 19  wherein the first confidence value is generated by evaluating motion vectors of graphical objects across the sequence of images. 
     
     
       23. The non-transitory machine-readable medium of  claim 22  further comprising:
 an image interpolator to generate the first image by interpolating from two or more other images in the sequence of images the image interpolator to use the motion vectors to perform the frame interpolation to generate the first image. 
 
     
     
       24. The non-transitory machine-readable medium of  claim 19  further comprising:
 a rendering engine to generate the sequence of images including the first image. 
 
     
     
       25. The non-transitory machine-readable medium of  claim 24  wherein the rendering engine comprises rasterization circuitry and/or logic. 
     
     
       26. The non-transitory machine-readable medium of  claim 24  wherein the rendering engine comprises ray tracing circuitry and/or logic. 
     
     
       27. The non-transitory machine-readable medium of  claim 19  wherein the first portion of the first image comprises N×N pixels and the first pixel patch comprises M×M pixels, where N<M.

Cited by (0)

No later patents cite this yet.

References (0)

No backward citations on record.