Methods and apparatus for foveated compression

Abstract

The present disclosure relates to methods and apparatus for graphics processing. Aspects of the present disclosure can render at least one frame including display content at a server. Aspects of the present disclosure can also downscale the at least one frame including the display content, where a downscaling rate of one or more portions of the at least one frame is based on a location of each of the one or more portions. Moreover, aspects of the present disclosure can communicate the downscaled at least one frame including the display content to a client device. Aspects of the present disclosure can also encode the downscaled at least one frame including the display content. Further, aspects of the present disclosure can decode the encoded at least one frame including the display content. Aspects of the present disclosure can also upscale the at least one frame including the display content.

What is claimed is: 
     
       1. A method of graphics processing, comprising:
 generating a foveated mesh for generating non-rectangular pulled-in frames; 
 rendering at least one frame including display content; 
 generating, based on the at least one frame and the foveated mesh, at least one non-rectangular pulled-in frame including the display content, wherein the at least one non-rectangular pulled-in frame is smaller in size than the at least one frame and comprises:
 at least one downscaled portion that is downscaled from the at least one frame; and 
 at least one preserved portion with pixels preserved at a 1:1 ratio from the at least one frame, wherein a size of the at least one preserved portion is determined in response to network constraints; 
 
 
       and
 communicating the at least one non-rectangular pulled-in frame including the display content to a client device. 
 
     
     
       2. The method of  claim 1 , further comprising:
 determining the downscaling rate of the at least one downscaled portion based on at least one of dimensions of the at least one frame, a field of view associated with the at least one frame, a fovea angle associated with the at least one frame, or one or more foveal regions associated with the at least one frame. 
 
     
     
       3. The method of  claim 1 , wherein the downscaling rate of the at least one downscaled portion is fixed over a period of time. 
     
     
       4. The method of  claim 1 , wherein the downscaling rate of the at least one downscaled portion is based on a quality of service (QOS) level for communication between a server and the client device. 
     
     
       5. The method of  claim 1 , wherein a location of the at least one downscaled portion and a location of the at least one preserved portion within the at least one non-rectangular pulled-in frame are based on an eye-tracked position of the client device. 
     
     
       6. The method of  claim 1 , wherein the downscaling rate of the at least one downscaled portion is based on a roundtrip delay associated with the rendering and the communicating. 
     
     
       7. The method of  claim 1 , wherein the downscaling rate of the at least one downscaled portion is based on a compression ratio between the at least one frame and the at least one non-rectangular pulled-in frame. 
     
     
       8. The method of  claim 1 , wherein the rendering is at varied resolutions across the at least one frame, and wherein the rendering is aligned with the downscaling. 
     
     
       9. The method of  claim 1 , further comprising:
 encoding the at least one non-rectangular pulled-in frame including the display content. 
 
     
     
       10. An apparatus for graphics processing, comprising:
 a memory; and 
 at least one processor coupled to the memory and configured to:
 generate a foveated mesh to generate non-rectangular pulled-in frames; 
 render at least one frame including display content; 
 generate, based on the at least one frame and the foveated mesh, at least one non-rectangular pulled-in frame including the display content, wherein the at least one non-rectangular pulled-in frame is smaller in size than the at least one frame and comprises:
 at least one downscaled portion that is downscaled from the at least one frame; and 
 at least one preserved portion with pixels preserved at a 1:1 ratio from the at least one frame, wherein a size of the at least one preserved portion is determined in response to network constraints; and 
 
 communicate the at least one non-rectangular pulled-in frame including the display content to a client device. 
 
 
     
     
       11. The apparatus of  claim 10 , wherein the at least one processor is further configured to determine the downscaling rate of the at least one downscaled portion based on at least one of dimensions of the at least one frame, a field of view associated with the at least one frame, a fovea angle associated with the at least one frame, or one or more foveal regions associated with the at least one frame. 
     
     
       12. The apparatus of  claim 10 , wherein the downscaling rate of the at least one downscaled portion is based on a quality of service (QOS) level for communication between a server and the client device. 
     
     
       13. The apparatus of  claim 10 , wherein the downscaling rate of the at least one downscaled portion is based on a roundtrip delay associated with the rendering and the communicating. 
     
     
       14. The apparatus of  claim 10 , wherein the downscaling rate of the at least one downscaled portion is based on a compression ratio between the at least one frame and the at least one non-rectangular pulled-in frame. 
     
     
       15. The apparatus of  claim 10 , wherein the at least one processor is further configured to render the at least one frame at varied resolutions across the at least one frame, and wherein the rendering is aligned with the downscaling. 
     
     
       16. The apparatus of  claim 10 , wherein the at least one processor is further configured to encode the at least one non-rectangular pulled-in frame including the display content. 
     
     
       17. A method of graphics processing, comprising:
 generating an inverted mesh corresponding to an inversion of a foveated mesh for applying to non-rectangular pulled-in frames; 
 receiving at least one non-rectangular pulled-in frame including display content, wherein the at least one non-rectangular pulled-in frame comprises:
 at least one downscaled portion that is downscaled from at least one original frame; and 
 at least one preserved portion with pixels preserved at a 1:1 ratio from the at least one original frame, wherein a size of the at least one preserved portion is determined in response to network constraints; 
 
 decoding the at least one non-rectangular pulled-in frame including the display content; and 
 increasing a size of the at least one non-rectangular pulled-in frame by applying the inverted mesh to the at least one non-rectangular pulled-in frame for at least upscaling the at least one downscaled portion and preserving a size of the at least one preserved portion. 
 
     
     
       18. The method of  claim 17 , wherein the upscaling rate of the at least one downscaled portion is fixed over a period of time. 
     
     
       19. The method of  claim 17 , wherein the upscaling rate of the at least one downscaled portion is based on a quality of service (QoS) level for communication between a server and a client device. 
     
     
       20. The method of  claim 17 , wherein a location of the at least one downscaled portion and a location of the at least one preserved portion within the at least one non-rectangular pulled-in frame are based on an eye-tracked position of a client device. 
     
     
       21. The method of  claim 17 , wherein the upscaling rate of the at least one downscaled portion is based on one of: a roundtrip delay associated with the receiving, or a compression ratio between the at least one non-rectangular pulled-in frame and the at least one original frame. 
     
     
       22. The method of  claim 17 , further comprising:
 warping the upscaled at least one non-rectangular pulled-in frame. 
 
     
     
       23. The method of  claim 22 , further comprising:
 receiving another frame including missing display content; and 
 concealing the missing display content with the warped, upscaled at least one non-rectangular pulled-in frame. 
 
     
     
       24. The method of  claim 23 , further comprising:
 downscaling the warped, upscaled at least one non-rectangular pulled-in frame prior to concealing the missing display content. 
 
     
     
       25. An apparatus for graphics processing, comprising:
 a memory; and 
 at least one processor coupled to the memory and configured to:
 generate an inverted mesh corresponding to an inversion of a foveated mesh to apply to non-rectangular pulled-in frames; 
 receive at least one non-rectangular pulled-in frame including display content, wherein the at least one non-rectangular pulled-in frame comprises:
 at least one downscaled portion that is downscaled from at least one original frame; and 
 at least one preserved portion with pixels preserved at a 1:1 ratio from the at least one original frame, wherein a size of the at least one preserved portion is determined in response to network constraints; 
 
 decode the at least one non-rectangular pulled-in frame including the display content; and 
 increase a size of the at least one non-rectangular pulled-in frame by application of the inverted mesh to the at least one non-rectangular pulled-in frame to at least upscale the at least one downscaled portion and preserve a size of the at least one preserved portion. 
 
 
     
     
       26. The apparatus of  claim 25 , wherein the upscaling rate of the at least one downscaled portion is based on a quality of service (QoS) level for communication between a server and a client device. 
     
     
       27. The apparatus of  claim 25 , wherein the upscaling rate of the at least one downscaled portion is based on one of: a roundtrip delay associated with the receiving, or a compression ratio between the at least one non-rectangular pulled-in frame and the at least one original frame. 
     
     
       28. The apparatus of  claim 25 , wherein the at least one processor is further configured to warp the upscaled at least one non-rectangular pulled-in frame. 
     
     
       29. The apparatus of  claim 28 , wherein the at least one processor is further configured to:
 receive another frame including missing display content; and 
 conceal the missing display content with the warped, upscaled at least one non-rectangular pulled-in frame. 
 
     
     
       30. The apparatus of  claim 29 , wherein the at least one processor is further configured to downscale the warped, upscaled at least one non-rectangular pulled-in frame prior to concealing the missing display content.