Systems and methods for deep-learning based super-resolution using multiple degradations on-demand learning

Abstract

A machine learning model can be trained using a first set of degraded images for each of a plurality of combinations and corresponding reference images, where a number of degraded images in the first set corresponding to a particular combination of the plurality of combinations is selected in accordance with a probability value associated with the particular combination. A validation process can be used to determine a loss value for each of the plurality of combinations of degradations. Updates to the probability values associated with the plurality of combinations can be calculated based on the loss values. The machine learning model can be updated using a second set of degraded images for each of the plurality of combinations, and the corresponding reference images, where a number of degraded images in the second set corresponding to the particular combination is selected based on the updated probability value.

What is claimed is: 
     
       1. A method comprising:
 obtaining a plurality of reference images; 
 applying a plurality of combinations of degradations on the reference images to generate degraded images corresponding to each of the plurality of combinations; 
 training a machine learning model using (i) a first set of degraded images for each of the plurality of combinations, and (ii) the corresponding reference images, wherein a number of degraded images in the first set corresponding to a particular combination of the plurality of combinations is selected in accordance with a probability value associated with the particular combination; 
 determining, using a validation process for the machine learning model, a loss value for each of the plurality of combinations of degradations; 
 calculating, based on the loss values, updates to the probability values associated with the plurality of combinations, to obtain updated probability values; and 
 updating the machine learning model using (i) a second set of degraded images for each of the plurality of combinations, and (ii) the corresponding reference images, wherein a number of degraded images in the second set corresponding to the particular combination is selected based on the updated probability value associated with the particular combination. 
 
     
     
       2. The method of  claim 1 , wherein the plurality of combinations of degradations comprise at least one of: noise, blur, or resolution downscaling. 
     
     
       3. The method of  claim 1 , wherein the probability values are initialized as equal to each other. 
     
     
       4. The method of  claim 1 , wherein the validation process comprises:
 obtaining a second plurality of reference images; and 
 applying the plurality of combinations of degradations on multiple subsets of the second plurality of reference images to generate validation images corresponding to each of the plurality of combinations. 
 
     
     
       5. The method of  claim 4 , wherein the validation process further comprises generating enhanced images from the validation images using the updated machine learning model. 
     
     
       6. The method of  claim 1 , wherein determining the loss value comprises at least one of: calculating a pixel loss value, a high-frequency loss value, a total loss value, or a match error. 
     
     
       7. The method of  claim 5 , wherein determining the loss value for a particular combination comprises determining one or more metrics of similarity between (i) validation images for the particular combination and (ii) the enhanced images. 
     
     
       8. The method of  claim 7 , wherein calculating updates to the probability values comprises:
 determining, that the loss value for a first particular combination is higher than the loss value for a second particular combination; and 
 responsive to determining that the loss value for the first particular combination is higher than the loss value for the second particular combination, assigning a first probability value to the first particular combination, the first probability value being higher than a second probability value assigned to the second particular combination. 
 
     
     
       9. A system, comprising:
 a computer-readable memory comprising computer-executable instructions; and 
 at least one processor executing the computer executable instructions to provide a machine learning module, wherein training of the machine learning module comprises:
 obtaining a plurality of reference images; 
 applying a plurality of combinations of degradations on the reference images to generate degraded images corresponding to each of the plurality of combinations; 
 training a machine learning model using (i) a first set of degraded images for each of the plurality of combinations, and (ii) the corresponding reference images, wherein a number of degraded images in the first set corresponding to a particular combination of the plurality of combinations is selected in accordance with a probability value associated with the particular combination; 
 determining, using a validation process for the machine learning model, a loss value for each of the plurality of combinations of degradations; 
 calculating, based on the loss values, updates to the probability values associated with the plurality of combinations, to obtain updated probability values; and 
 updating the machine learning model using (i) a second set of degraded images for each of the plurality of combinations, and (ii) the corresponding reference images, wherein a number of degraded images in the second set corresponding to the particular combination is selected based on the updated probability value associated with the particular combination. 
 
 
     
     
       10. The system of  claim 9 , wherein the plurality of combinations of degradations comprise at least one of: noise, blur, or resolution downscaling. 
     
     
       11. The system of  claim 9 , wherein the probability values are initialized as equal to each other. 
     
     
       12. The system of  claim 9 , wherein the validation process comprises:
 obtaining a second plurality of reference images; and 
 applying the plurality of combinations of degradations on multiple subsets of the second plurality of reference images to generate validation images corresponding to each of the plurality of combinations. 
 
     
     
       13. The system of  claim 12 , wherein the validation process further comprises generating enhanced images from the validation images using the updated machine learning model. 
     
     
       14. The system of  claim 9 , wherein determining the loss value comprises at least one of: calculating a pixel loss value, a high-frequency loss value, a total loss value, or a match error. 
     
     
       15. The system of  claim 14 , wherein determining the loss value for a particular combination comprises determining one or more metrics of similarity between (i) validation images for the particular combination and (ii) enhanced images. 
     
     
       16. One or more non-transitory machine-readable storage devices encoded with instructions configured to cause one or more processing devices to execute operations comprising:
 obtaining a plurality of reference images; 
 applying a plurality of combinations of degradations on the reference images to generate degraded images corresponding to each of the plurality of combinations; 
 training a machine learning model using (i) a first set of degraded images for each of the plurality of combinations, and (ii) the corresponding reference images, wherein a number of degraded images in the first set corresponding to a particular combination of the plurality of combinations is selected in accordance with a probability value associated with the particular combination; 
 determining, using a validation process for the machine learning model, a loss value for each of the plurality of combinations of degradations; 
 calculating, based on the loss values, updates to the probability values associated with the plurality of combinations, to obtain updated probability values; and 
 updating the machine learning model using (i) a second set of degraded images for each of the plurality of combinations, and (ii) the corresponding reference images, wherein a number of degraded images in the second set corresponding to the particular combination is selected based on the updated probability value associated with the particular combination. 
 
     
     
       17. The one or more non-transitory machine-readable storage devices of  claim 16 , wherein the plurality of combinations of degradations comprise at least one of: noise, blur, or resolution downscaling. 
     
     
       18. The one or more non-transitory machine-readable storage devices of  claim 16 , wherein the validation process comprises:
 obtaining a second plurality of reference images; and 
 applying the plurality of combinations of degradations on multiple subsets of the second plurality of reference images to generate validation images corresponding to each of the plurality of combinations. 
 
     
     
       19. The one or more non-transitory machine-readable storage devices of  claim 18 , wherein determining the loss value for a particular combination comprises determining one or more metrics of similarity between (i) validation images for the particular combination and (ii) the enhanced images. 
     
     
       20. The one or more non-transitory machine-readable storage devices of  claim 19 , wherein calculating updates to the probability values comprises:
 determining, that the loss value for a first particular combination is higher than the loss value for a second particular combination; and 
 responsive to determining that the loss value for the first particular combination is higher than the loss value for the second particular combination, assigning a first probability value to the first particular combination, the first probability value being higher than a second probability value assigned to the second particular combination.