Foveated Rendering

Abstract

Foveated rendering for rendering an image uses a ray tracing technique to process graphics data for a region of interest of the image, and a rasterisation technique is used to process graphics data for other regions of the image. A rendered image can be formed using the processed graphics data for the region of interest of the image and the processed graphics data for the other regions of the image. The region of interest may correspond to a foveal region of the image. Ray tracing naturally provides high detail and photo-realistic rendering, which human vision is particularly sensitive to in the foveal region; whereas rasterisation techniques are suited for providing temporal smoothing and anti-aliasing in a simple manner, and is therefore suited for use in the regions of the image that a user will see in the periphery of their vision.

What is claimed is: 
     
         1 . A processing system configured to render one or more images of a scene, the processing system comprising:
 rendering logic configured to process graphics data to generate an initial image comprising pixel values representing an image of the scene;   ray tracing logic configured to perform ray tracing to determine ray traced data for one or more regions of the initial image; and   update logic configured to update the initial image using the determined ray traced data for the one or more regions of the initial image, to thereby determine an updated image to be outputted for display.   
     
     
         2 . The processing system of  claim 1 , wherein the rendering logic is configured to process the graphics data using a rasterisation technique to generate the initial image. 
     
     
         3 . The processing system of  claim 1 , wherein the rendering logic is configured to process the graphics data using a ray tracing technique to generate the initial image. 
     
     
         4 . The processing system of  claim 1 , wherein the initial image is a lower detail image than the updated image. 
     
     
         5 . The processing system of  claim 1 , further comprising region identification logic configured to identify the one or more regions of the initial image and gaze tracking logic configured to determine one or more gaze positions for the initial image, wherein the region identification logic is configured to receive one or more indications of the one or more determined gaze positions, and to identify the one or more regions of the initial image based on the one or more determined gaze positions. 
     
     
         6 . The processing system of  claim 5 , wherein the gaze tracking logic is configured to implement a predictive model to anticipate movements in gaze. 
     
     
         7 . The processing system of  claim 5 , wherein one of the one or more identified regions of the initial image surrounds one of the one or more determined gaze positions, thereby representing a foveal region. 
     
     
         8 . The processing system of  claim 7 , further comprising a camera pipeline which is configured to:
 receive image data from a camera which is arranged to capture images of a user looking at a display on which a rendered image is to be displayed; and   process the received image data to generate a captured image;   wherein the gaze tracking logic is configured to analyse the captured image to determine the gaze position for the initial image.   
     
     
         9 . The processing system of  claim 8 , wherein the ray tracing logic and the rasterisation logic are implemented on a graphics processing unit, and wherein the camera pipeline and the graphics processing unit are implemented as part of a system on chip (SOC). 
     
     
         10 . The processing system of  claim 1 , further comprising region identification logic configured to identify the one or more regions of the initial image, wherein the region identification logic is configured to analyse the initial image to determine one or more regions of high frequency, wherein the one or more determined regions of high frequency are the one or more identified regions of the initial image. 
     
     
         11 . The processing system of  claim 1 , wherein the rendering logic and the ray tracing logic are configured to operate asynchronously. 
     
     
         12 . The processing system of  claim 1 , wherein the ray tracing logic and the update logic are configured to operate at a first rate, and wherein the rendering logic is configured to operate at a second rate, wherein the first rate is faster than the second rate. 
     
     
         13 . The processing system of  claim 11 , further comprising time warping logic configured to apply an image warping process to the updated image before it is sent for display. 
     
     
         14 . The processing system of  claim 1 , further comprising acceleration structure building logic configured to determine an acceleration structure representing the graphics data of geometry in a scene of which an image is to be rendered. 
     
     
         15 . The processing system of  claim 14 , wherein the processing system is configured to render a plurality of images representing a sequence of frames, and wherein the acceleration structure building logic is configured to determine the acceleration structure for a current frame by updating the acceleration structure for the preceding frame. 
     
     
         16 . The processing system of  claim 1 , wherein the processing system is arranged to be included in a virtual reality system or an augmented reality system. 
     
     
         17 . The processing system of  claim 1 , wherein the update logic is configured to update the initial image using the determined ray traced data for the one or more regions of the initial image by adding detail to the one or more regions of the initial image. 
     
     
         18 . A method of rendering one or more images of a scene at a processing system, the method comprising:
 processing graphics data to generate an initial image comprising pixel values representing an image of the scene;   performing ray tracing to determine ray traced data for one or more regions of the initial image; and   updating the initial image using the determined ray traced data for the one or more regions of the initial image, to thereby determine an updated image to be outputted for display.   
     
     
         19 . The method of  claim 18 , further comprising displaying an image based on the updated image. 
     
     
         20 . A non-transitory computer readable storage medium having stored thereon a computer readable dataset description of an integrated circuit that, when processed in an integrated circuit manufacturing system, causes the integrated circuit manufacturing system to manufacture a processing system comprising:
 rendering logic configured to process graphics data to generate an initial image comprising pixel values representing an image of a scene;   ray tracing logic configured to perform ray tracing to determine ray traced data for one or more regions of the initial image; and   update logic configured to update the initial image using the determined ray traced data for the one or more regions of the initial image, to thereby determine an updated image to be outputted for display.