Display apparatuses and rendering servers incorporating prioritized re-rendering

Abstract

A display apparatus including means for tracking pose of user's head, light source(s) and processor configured to: process pose-tracking data to determine position, orientation, velocity and acceleration of head; predict viewpoint and view direction of user in extended-reality environment; determine region of extended-reality environment to be presented, based on viewpoint and view direction; determine sub-region(s) of region whose rendering information is to be derived from previous rendering information of corresponding sub-region(s) of previously-presented region of extended-reality environment; generate rendering information of sub-region(s) based on previous rendering information; send, to rendering server, information indicating remaining sub-regions required to be re-rendered and pose information indicating viewpoint and view direction; receive, from rendering server, rendering information of remaining sub-regions; merge rendering information of sub-region(s) and rendering information of remaining sub-regions to generate image(s); and display image(s) via light source(s).

What is claimed is: 
     
       1. A display apparatus comprising:
 means for tracking a pose of a user's head; 
 at least one light source per eye; and 
 at least one processor configured to: 
 process pose-tracking data, collected by said means, to determine a position, an orientation, a velocity and an acceleration of the user's head;
 predict a viewpoint and a view direction of the user in an extended-reality environment, based on the position, the orientation, the velocity and the acceleration of the user's head; 
 determine a region of the extended-reality environment that is to be presented to the user, based on the predicted viewpoint and the predicted view direction of the user; 
 determine at least one sub-region of the region to be presented whose rendering information is to be reused from previous rendering information of at least one corresponding sub-region of at least one previously-presented region of the extended-reality environment; 
 generate the rendering information of the at least one sub-region based on the previous rendering information; 
 send, to a rendering server, information indicative of remaining sub-regions of the region to be presented that have a required resolution greater than the resolution of the remaining sub region's previous rendering information and are required to be re-rendered by the rendering server, the information indicative of the remaining sub-regions including pose information indicative of the predicted viewpoint and the predicted view direction of the user; 
 receive, from the rendering server, rendering information of the re-rendered remaining sub-regions of the region to be presented; 
 merge the rendering information of the at least one sub-region and the rendering information of the remaining sub-regions to generate at least one image; and 
 display the at least one image via the at least one light source. 
 
 
     
     
       2. The display apparatus of  claim 1 , further comprising means for tracking a user's gaze, wherein the at least one processor is configured to:
 process gaze-tracking data, collected by the means for tracking the user's gaze, to determine a gaze direction, a gaze velocity and a gaze acceleration per eye of the user; 
 predict a gaze location of the user in the extended-reality environment, based on the gaze direction, the gaze velocity and the gaze acceleration; and 
 send, to the rendering server, gaze information indicative of the predicted gaze location, 
 wherein the region to be presented is determined based on the predicted gaze location. 
 
     
     
       3. The display apparatus of  claim 2 , wherein the at least one sub-region of the region to be presented whose rendering information is to be derived from the previous rendering information is determined based on the predicted gaze location. 
     
     
       4. The display apparatus of  claim 1 , wherein the at least one processor is configured to compute a quality score for each sub-region of the region to be presented, wherein a quality score is computed for a given sub-region based on whether or not previous rendering information of at least one corresponding sub-region of the at least one previously-presented region is available for the given sub-region. 
     
     
       5. The display apparatus of  claim 4 , wherein the quality score is computed for the given sub-region based on a comparison of a resolution required for the given sub-region and a resolution at which the previous rendering information of the at least one corresponding sub-region is available. 
     
     
       6. The display apparatus of  claim 4 , wherein the at least one processor is configured to:
 determine a priority order in which the remaining sub-regions of the region to be presented are required to be re-rendered, based on corresponding quality scores of the remaining sub-regions; and 
 send the priority order to the rendering server, 
 wherein the rendering information of the remaining sub-regions is received from the rendering server as per the priority order. 
 
     
     
       7. The display apparatus of  claim 4 , wherein, when determining the at least one sub-region of the region to be presented whose rendering information is to be derived from the previous rendering information, the at least one processor is configured to identify the given sub-region as one of the at least one sub-region when the quality score of the given sub-region is above a predefined threshold score. 
     
     
       8. A method of displaying, the method being implemented via a display apparatus comprising means for tracking a pose of a user's head and at least one light source per eye, the method comprising:
 processing pose-tracking data, collected by said means, to determine a position, an orientation, a velocity and an acceleration of the user's head; 
 predicting a viewpoint and a view direction of the user in an extended-reality environment, based on the position, the orientation, the velocity and the acceleration of the user's head; 
 determining a region of the extended-reality environment that is to be presented to the user, based on the predicted viewpoint and the predicted view direction of the user; 
 determining at least one sub-region of the region to be presented whose rendering information is to be reused from previous rendering information of at least one corresponding sub-region of at least one previously-presented region of the extended-reality environment; 
 generating the rendering information of the at least one sub-region based on the previous rendering information; 
 sending, to a rendering server, information indicative of remaining sub-regions of the region to be presented that have a required resolution greater than the resolution of the remaining sub region's previous rendering information and are required to be re-rendered by the rendering server, the information indicative of the remaining sub-regions including pose information indicative of the predicted viewpoint and the predicted view direction of the user; 
 receiving, from the rendering server, rendering information of the re-rendered remaining sub-regions of the region to be presented; 
 merging the rendering information of the at least one sub-region and the rendering information of the remaining sub-regions to generate at least one image; and 
 displaying the at least one image via the at least one light source. 
 
     
     
       9. The method of  claim 8 , wherein the display apparatus further comprises means for tracking a user's gaze, and wherein the method further comprises:
 processing gaze-tracking data, collected by the means for tracking the user's gaze, to determine a gaze direction, a gaze velocity and a gaze acceleration per eye of the user; 
 predicting a gaze location of the user in the extended-reality environment, based on the gaze direction, the gaze velocity and the gaze acceleration; and 
 sending, to the rendering server, gaze information indicative of the predicted gaze location, 
 wherein the region to be presented is determined based on the predicted gaze location. 
 
     
     
       10. The method of  claim 9 , wherein the at least one sub-region of the region to be presented whose rendering information is to be derived from the previous rendering information is determined based on the predicted gaze location. 
     
     
       11. The method of  claim 8 , further comprising computing a quality score for each sub-region of the region to be presented, wherein a quality score is computed for a given sub-region based on whether or not previous rendering information of at least one corresponding sub-region of the at least one previously-presented region is available for the given sub-region. 
     
     
       12. The method of  claim 11 , wherein the quality score is computed for the given sub-region based on a comparison of a resolution required for the given sub-region and a resolution at which the previous rendering information of the at least one corresponding sub-region is available. 
     
     
       13. The method of  claim 11 , further comprising:
 determining a priority order in which the remaining sub-regions of the region to be presented are required to be re-rendered, based on corresponding quality scores of the remaining sub-regions; and 
 sending the priority order to the rendering server, 
 wherein the rendering information of the remaining sub-regions is received from the rendering server as per the priority order. 
 
     
     
       14. The method of  claim 11 , wherein the step of determining the at least one sub-region of the region to be presented whose rendering information is to be derived from the previous rendering information comprises identifying the given sub-region as one of the at least one sub-region when the quality score of the given sub-region is above a predefined threshold score.