Device, system and method for content-adaptive resolution-enhancement

Abstract

A device, system and method for content-adaptive resolution-enhancement is provided. A plurality of subframe streams are generated from a video stream, each of the plurality of subframe streams comprising a lower resolution version of the video stream, pixel-shifted from one another. A plurality of output subframe streams are generated from the plurality of subframe streams in a one-to-one relationship by: applying a plurality of video enhancement filters to each of the plurality of subframe streams, each of the plurality of video enhancement filters for enhancing different features of the video stream; and, combining one or more resulting enhanced subframe streams into a respective output subframe stream based on data in one or more regions of the video stream. One or more projectors are controlled to project the plurality of output subframe streams to combine the plurality of output subframe streams into a higher resolution projected video stream.

What is claimed is: 
     
       1. A device comprising:
 a controller and a communication interface configured to communicate with one or more projectors, the controller configured to: 
 generate a plurality of subframe streams from a video stream; each of the plurality of subframe streams comprising a lower resolution version of the video stream, pixel-shifted from one another; 
 generate a plurality of output subframe streams from the plurality of subframe streams in a one-to-one relationship by: applying a plurality of video enhancement filters to each of the plurality of subframe streams by: converting each of the plurality of subframe streams from a spatial domain to a frequency domain; applying a respective video enhancement filter in the frequency domain; and converting the respective output subframe stream back to the spatial domain, each of the plurality of video enhancement filters for enhancing different features of the video stream; and, combining one or more resulting enhanced subframe streams into a respective output subframe stream based on data in one or more regions of the video stream; and, 
 control, using the communication interface, the one or more projectors to project the plurality of output subframe streams, thereby combining the plurality of output subframe streams into a higher resolution projected video stream, wherein: the plurality of video enhancement filters comprises: a first enhancement video filter for enhancing moving objects in the video stream, and a second enhancement video filter for enhancing static objects in the video stream, and the one or more resulting enhanced subframe stream comprises a first enhanced subframe stream for the moving objects, and a second enhanced subframe stream enhanced for the static objects; 
 combine the first enhanced subframe stream and the second enhanced subframe stream into the respective output subframe stream based on the data in the one or more regions of the video stream by: determining respective regions where the moving objects and the static objects are located in the video stream; and including corresponding portion of the first enhanced subframe stream in moving object regions and including corresponding portions of the second enhanced subframe stream in static objects regions. 
 
     
     
       2. The device of  claim 1 , wherein the controller is further configured to generate the plurality of subframe streams by one more of resampling, upsampling and downsampling the video stream. 
     
     
       3. The device of  claim 1 , wherein the controller is further configured to apply one or more of the plurality of video enhancement filters to each of the plurality of subframe streams by: applying a further respective video enhancement filter in a spatial domain. 
     
     
       4. The device of  claim 1 , wherein the plurality of video enhancement filters comprises: a first enhancement video filter for enhancing moving objects in the video stream, and a second enhancement video filter for enhancing static objects in the video stream. 
     
     
       5. The device of  claim 4 , wherein the one or more resulting enhanced subframe streams comprises a first enhanced subframe stream enhanced for the moving objects, and a second enhanced subframe stream enhanced for the static objects. 
     
     
       6. The device of  claim 5 , wherein the controller is further configured to combine the first enhanced subframe stream and the second enhanced subframe stream into the respective output subframe stream based on the data in the one or more regions of the video stream by: determining respective regions where the moving objects and the static objects are located in the video stream; and including corresponding portions of the first enhanced subframe stream in moving object regions and including corresponding portions of the second enhanced subframe stream in static object regions. 
     
     
       7. The device of  claim 1 , wherein the controller is further configured to determine the data in the one or more regions of the video stream by comparing successive frames of the video stream. 
     
     
       8. The device of  claim 1 , wherein the plurality of video enhancement filters comprises one or more of: a moving object video enhancement filter, a static object video enhancement filter, a text enhancement filter, a texture enhancement filter, and a color enhancement filter. 
     
     
       9. The device of  claim 1 , wherein the controller is further configured to apply a compensation filter to each of respective enhanced subframe streams, the compensation filter for compensating for optical aberrations of the one or more projectors. 
     
     
       10. A method comprising:
 at a device configured to communicate with one or more projectors, generating, at the device, a plurality of subframe streams from a video stream; each of the plurality of subframe streams comprising a lower resolution version of the video stream, pixel-shifted from one another; 
 generating, at the device, a plurality of output subframe streams from the plurality of subframe streams in a one-to-one relationship by: applying a plurality of video enhancement filters to each of the plurality of subframe streams by: converting each of the plurality of subframe streams from a spatial domain to a frequency domain; applying a respective video enhancement filter in the frequency domain; and converting the respective output subframe stream back to the spatial domain, each of the plurality of video enhancement filters for enhancing different features of the video stream; and, combining one or more resulting enhanced subframe streams into a respective output subframe stream based on data in one or more regions of the video stream; and, 
 controlling, using the device, the one or more projectors to project the plurality of output subframe streams, thereby combining the plurality of output subframe streams into a higher resolution projected video stream, wherein: the plurality of video enhancement filters comprises: a first enhancement video filter for enhancing moving objects in the video stream, and a second enhancement video filter for enhancing static objects in the video stream, the one or more resulting enhanced subframe streams comprises a first enhanced subframe stream enhanced for the moving objects, and a second enhanced Subframe stream enhanced for the static objects; and 
 Combining at the device, the first enhance subframe stream and the second enhanced subframe stream into the respective output subframe stream based on the data in the one or more regions of the video stream by: determining respective regions where the moving objects and the static objects are located in the video stream; and including corresponding portions of the first enhanced subframe stream in moving object regions and including corresponding portions of the second enhanced subframe stream in static object regions. 
 
     
     
       11. The method of  claim 10 , further comprising generating, at the device, the plurality of subframe streams by one more of resampling, upsampling and downsampling the video stream. 
     
     
       12. The method of  claim 10 , further comprising applying, at the device, one or more of the plurality of video enhancement filters to each of the plurality of subframe streams by: applying a further respective video enhancement filter in a spatial domain. 
     
     
       13. The method of  claim 10 , wherein the plurality of video enhancement filters comprises: a first enhancement video filter for enhancing moving objects in the video stream, and a second enhancement video filter for enhancing static objects in the video stream. 
     
     
       14. The method of  claim 13 , wherein the one or more resulting enhanced subframe streams comprises a first enhanced subframe stream enhanced for the moving objects, and a second enhanced subframe stream enhanced for the static objects, and the method further comprising combining, at the device, the first enhanced subframe stream and the second enhanced subframe stream into the respective output subframe stream based on the data in the one or more regions of the video stream by: determining respective regions where the moving objects and the static objects are located in the video stream; and including corresponding portions of the first enhanced subframe stream in moving object regions and including corresponding portions of the second enhanced subframe stream in static object regions. 
     
     
       15. The method of  claim 10 , further comprising determining, at the device, the data in the one or more regions of the video stream by comparing successive frames of the video stream. 
     
     
       16. The method of  claim 10 , wherein the plurality of video enhancement filters comprises one or more of: a moving object video enhancement filter, a static object video enhancement filter, a text enhancement filter, a texture enhancement filter, and a color enhancement filter. 
     
     
       17. The method of  claim 10 , further comprising applying, at the device, a compensation filter to each of respective enhanced subframe streams, the compensation filter for compensating for optical aberrations of the one or more projectors. 
     
     
       18. A non-transitory computer-readable medium storing a computer program, wherein execution of the computer program is for:
 at a device configured to communicate with one or more projectors, generating, at the device, a plurality of subframe streams from a video stream; each of the plurality of subframe streams comprising a lower resolution version of the video stream, pixel-shifted from one another; 
 generating, at the device, a plurality of output subframe streams from the plurality of subframe streams in a one-to-one relationship by: applying a plurality of video enhancement filters to each of the plurality of subframe streams by: converting each of the plurality of subframe streams from a spatial domain to a frequency domain; applying a respective video enhancement filter in the frequency domain; and converting the respective output subframe stream back to the spatial domain, each of the plurality of video enhancement filters for enhancing different features of the video stream; and, combining one or more resulting enhanced subframe streams into a respective output subframe stream based on data in one or more regions of the video stream; and, 
 controlling, using the device, the one or more projectors to project the plurality of output subframe streams, thereby combining the plurality of output subframe streams into a higher resolution projected video stream, wherein, the plurality of video enhancement filter comprises a first enhancement video filter for enhancing moving objects in the video stream, and second enhancement video filter for enhancing static objects in the video stream, the one or more resulting enhanced subframe streams comprises a first enhanced subframe stream enhanced for the moving objects, and a second enhanced subframe stream enhanced for the static objects; and 
 Combing, at the device, the first enhanced subframe stream and the second enhanced subframe stream into the respective output subframe stream based on the data in the one or more regions of the video stream by, determining respective regions where the moving objects and the static objects are located in the video stream; and including corresponding portions of the first enhanced subframe stream in moving object regions and including corresponding portions of the second enhanced, subframe stream in static object regions. 
 
     
     
       19. The non-transitory computer-readable medium of claim 18, wherein execution of the computer program further comprises generating, at the device, the plurality of subframe streams by one more of resampling, upsampling and downsampling the video stream. 
     
     
       20. The non-transitory computer-readable medium of claim 18, wherein execution of the computer program further comprises applying, at the device, one or more of the plurality of video enhancement filters to each of the plurality of subframe streams by applying a further respective video enhancement filter in a spatial domain.