Image Compression and Reconstruction Using Machine Learning Models

Abstract

A method includes obtaining image data, identifying a machine learning-compressible (ML-compressible) portion of the image data, and determining a location of the ML-compressible portion within the image data. The method also includes selecting, from a plurality of ML compression models, an ML compression model for the ML-compressible portion based on an image content thereof, and generating, based on the ML-compressible portion and by the ML compression model, an ML-compressed representation of the ML-compressible portion. The method further includes generating a compressed image data file that includes the ML-compressed representation and the location of the ML-compressible portion, and outputting the compressed image data file. The compressed image data file is configured to cause an ML decompression model corresponding to the ML compression model to generate a reconstruction of the ML-compressible portion of the image data based on the ML-compressed representation.

1 . A computer-implemented method comprising:
 obtaining image data;   identifying a machine learning-compressible (ML-compressible) portion of the image data and determining a location of the ML-compressible portion within the image data;   selecting, from a plurality of machine learning (ML) compression models, an ML compression model for the ML-compressible portion of the image data based on an image content thereof;   generating, based on the ML-compressible portion of the image data and by the ML compression model, an ML-compressed representation of the ML-compressible portion of the image data;   generating a compressed image data file comprising the ML-compressed representation and the location of the ML-compressible portion, wherein the compressed image data file is configured to cause an ML decompression model corresponding to the ML compression model to generate a reconstruction of the ML-compressible portion of the image data based on the ML-compressed representation; and   outputting the compressed image data file.   
     
     
         2 . The computer-implemented method of  claim 1 , wherein one or more of (i) a frequency content of the reconstruction of the ML-compressible portion or (ii) a resolution of the reconstruction of the ML-compressible portion is substantially lossless with respect to the ML-compressible portion, and wherein a visual accuracy of the reconstruction of the ML-compressible portion is lossy with respect to the ML-compressible portion. 
     
     
         3 . The computer-implemented method of  claim 1 , further comprising:
 identifying a non-ML-compressible portion of the image data and determining a location of the non-ML-compressible portion within the image data, wherein the compressed image data file further comprises the non-ML-compressible portion and the location of the non-ML-compressible portion, and wherein the compressed image data file is configured to provide for reconstructing the image data by compositing the reconstruction of the ML-compressible portion of the image data with the non-ML-compressible portion of the image data.   
     
     
         4 . The computer-implemented method of  claim 1 , wherein each respective ML compression model of the plurality of ML compression models is configured to generate ML-compressed representations for a corresponding type of image content, and wherein selecting the ML compression model comprises:
 determining a type of the image content of the ML-compressible portion; and   based on the type of the image content of the ML-compressible portion, selecting the ML compression model from the plurality of ML compression models.   
     
     
         5 . The computer-implemented method of  claim 1 , wherein identifying the ML-compressible portion of the image data and determining the location of the ML-compressible portion within the image data comprises:
 generating, by a segmentation model, a segmentation mask indicating pixels of the image data that represent the ML-compressible portion of the image data.   
     
     
         6 . The computer-implemented method of  claim 1 , wherein identifying the ML-compressible portion of the image data and determining the location of the ML-compressible portion within the image data comprises:
 dividing the image data into a grid comprising a plurality of cells each comprising a respective plurality of pixels;   determining, for each respective cell of the plurality of cells, whether the respective plurality of pixels represents image content that is ML-compressible by at least one ML compression model of the plurality of ML compression models; and   identifying the ML-compressible portion based on determining that the respective plurality of pixels of a particular cell of the plurality of cells represents the image content that is ML-compressible by the ML compression model.   
     
     
         7 . The computer-implemented method of  claim 1 , wherein identifying the ML-compressible portion of the image data and determining the location of the ML-compressible portion within the image data comprises:
 displaying, by way of a user interface, the image data; and   receiving, by way of the user interface, a manual selection of the ML-compressible portion from the image data as displayed.   
     
     
         8 . The computer-implemented method of  claim 1 , wherein:
 identifying the ML-compressible portion of the image data comprises dividing the image data into a plurality of semantically-distinct ML-compressible portions;   determining the location of the ML-compressible portion within the image data comprises determining, for each respective ML-compressible portion of the plurality of semantically-distinct ML-compressible portions, a corresponding location of the respective ML-compressible portion;   selecting the ML compression model comprises selecting, for each respective ML-compressible portion of the plurality of semantically-distinct ML-compressible portions and from the plurality of ML compression models, a corresponding ML compression model based on a respective image content of the respective ML-compressible portion;   generating the ML-compressed representation comprises generating, for each respective ML-compressible portion of the plurality of semantically-distinct ML-compressible portions and by the corresponding ML compression model, a corresponding ML-compressed representation of the respective ML-compressible portion based on the respective ML-compressible portion;   the compressed image data file comprises, for each respective ML-compressible portion of the plurality of semantically-distinct ML-compressible portions, the corresponding ML-compressed representation and the corresponding location; and   each respective ML compression model of the plurality of ML compression models is associated with a corresponding ML decompression model configured to generate reconstructions of ML-compressible portions of image data based on ML-compressed representations generated by the respective ML compression model.   
     
     
         9 . The computer-implemented method of  claim 1 , wherein the ML-compressible portion is identified based on an importance of the image content of the ML-compressible portion, and wherein a variability of the reconstruction of the ML-compressible portion is inversely proportional to the importance of the image content. 
     
     
         10 . The computer-implemented method of  claim 1 , wherein the ML-compressed representation comprises one or more of: (i) a vector representing the image content of the ML-compressible portion of the image data or (ii) a textual string describing the image content of the ML-compressible portion of the image data, wherein the ML decompression model is configured to generate the reconstruction of the ML-compressible portion of the image data based on the one or more of: (i) the vector or (ii) the textual string. 
     
     
         11 . The computer-implemented method of  claim 10 , wherein an image database is configured to store a plurality of compressed image data files corresponding to a plurality of image data, and wherein the computer-implemented method further comprises:
 compressing, using a text compression algorithm and for each respective compressed image data file of the plurality of compressed image data files, a corresponding textual string of the respective compressed image data file, wherein the respective compressed image data file is configured to store the corresponding textual string.   
     
     
         12 . The computer-implemented method of  claim 1 , further comprising:
 receiving the compressed image data file;   generating, based on the ML-compressed representation and by the ML decompression model, the reconstruction of the ML-compressible portion of the image data; and   generating decompressed image data by positioning, within the decompressed image data, the reconstruction of the ML-compressible portion according to the location of the ML-compressible portion.   
     
     
         13 . The computer-implemented method of  claim 12 , wherein generating the reconstruction of the ML-compressible portion of the image data comprises:
 identifying, within an image database, one or more reference image data based on a similarity between the ML-compressed representation and respective ML-compressed representations of the one or more reference image data; and   generating the reconstruction of the ML-compressible portion further based on respective image content of the one or more reference image data.   
     
     
         14 . The computer-implemented method of  claim 12 , wherein generating the reconstruction of the ML-compressible portion of the image data comprises:
 receiving a request to modify an attribute of the ML-compressible portion, wherein the attribute is represented by the ML-compressed representation;   generating an adjusted ML-compressed representation by modifying a value of the ML-compressed representation; and   generating the reconstruction of the ML-compressible portion based on the adjusted ML-compressed representation.   
     
     
         15 . The computer-implemented method of  claim 12 , wherein generating the reconstruction of the ML-compressible portion of the image data comprises:
 generating, by the ML decompression model, a plurality of different reconstructions of the ML-compressible portion of the image data;   displaying the plurality of different reconstructions of the ML-compressible portion of the image data; and   receiving a selection of a particular reconstruction from the plurality of different reconstructions, wherein the decompressed image data is generated based on the particular reconstruction.   
     
     
         16 . The computer-implemented method of  claim 1 , wherein the compressed image data file further comprises image attribute data comprising one or more of: (i) a time at which the image data has been captured, (ii) weather conditions at the time at which the image data has been captured, (iii) a geographic location associated with the image data, (iv) one or more parameters of a camera used to capture the image data, or (v) sensor data generated by one or more sensors on the camera used to capture the image data, and wherein the ML decompression model is configured to generate the reconstruction further based on the image attribute data. 
     
     
         17 . The computer-implemented method of  claim 1 , wherein the image data comprises a plurality of image frames that form a video, wherein the ML-compressible portion comprises (i) a first ML-compressible portion located at a first location of a first image frame of the plurality of image frames and (ii) a second ML-compressible portion located at a second location of a second image frame of the plurality of image frames, wherein the first ML-compressible portion and the second ML-compressible portion each represent a same image content at different respective times, wherein the ML-compressed representation comprises a first ML-compressed representation of the first ML-compressible portion and a second ML-compressed representation of the second ML-compressible portion, and wherein the ML decompression model is configured to generate a first reconstruction of the first ML-compressible portion based on the first ML-compressed representation and a second reconstruction of the second ML-compressible portion based on the second ML-compressed representation. 
     
     
         18 . The computer-implemented method of  claim 17 , further comprising:
 generating, by the ML decompression model, the first reconstruction based on the first ML-compressed representation and the second reconstruction based on the second ML-compressed representation;   generating (i) a first decompressed image frame by positioning, within the first decompressed image frame, the first reconstruction according to the first location and (ii) a second decompressed image frame by positioning, within the second decompressed image frame, the second reconstruction according to the second location; and   generating, by a video interpolation model and based on the first decompressed image frame and the second decompressed image frame, an interpolated image frame positioned between the first decompressed image frame and the second decompressed image frame within the video.   
     
     
         19 . A system comprising:
 a processor; and   a non-transitory computer-readable medium having stored thereon instructions that, when executed by the processor, cause the processor to perform operations comprising:
 obtaining image data; 
 identifying a machine learning-compressible (ML-compressible) portion of the image data and determining a location of the ML-compressible portion within the image data; 
 selecting, from a plurality of machine learning (ML) compression models, an ML compression model for the ML-compressible portion of the image data based on an image content thereof; 
   generating, based on the ML-compressible portion of the image data and by the ML compression model, an ML-compressed representation of the ML-compressible portion of the image data;   generating a compressed image data file comprising the ML-compressed representation and the location of the ML-compressible portion, wherein the compressed image data file is configured to cause an ML decompression model corresponding to the ML compression model to generate a reconstruction of the ML-compressible portion of the image data based on the ML-compressed representation; and   outputting the compressed image data file.   
     
     
         20 . A non-transitory computer-readable medium having stored thereon instructions that, when executed by a computing device, cause the computing device to perform operations comprising:
 obtaining image data;   identifying a machine learning-compressible (ML-compressible) portion of the image data and determining a location of the ML-compressible portion within the image data;   selecting, from a plurality of machine learning (ML) compression models, an ML compression model for the ML-compressible portion of the image data based on an image content thereof;   generating, based on the ML-compressible portion of the image data and by the ML compression model, an ML-compressed representation of the ML-compressible portion of the image data;   generating a compressed image data file comprising the ML-compressed representation and the location of the ML-compressible portion, wherein the compressed image data file is configured to cause an ML decompression model corresponding to the ML compression model to generate a reconstruction of the ML-compressible portion of the image data based on the ML-compressed representation; and   outputting the compressed image data file.