Semantic image retrieval for whole slide images

Abstract

Presented herein are systems and methods for semantic image retrieval. A computing system may identify a first biomedical image. The computing system may apply an image retrieval model to the first biomedical image. The image retrieval model may have a convolution block having a first plurality of parameters to generate a feature map using the first biomedical image. The first plurality of parameters may be transferred from a preliminary model. The image retrieval model may have an encoder having a second plurality of parameters to generate a first hash code for the first biomedical image based on the feature map. The computing system may select. from the plurality of second biomedical images corresponding to a plurality of second hash codes, a subset of second biomedical images using the first hash code. The computing system may provide the subset of second biomedical images identified using the first biomedical image.

What is claimed is: 
     
         1 . A method, comprising:
 establishing, by a computing system, a first model comprising a first plurality of kernel parameters, using a first training dataset having a plurality of unlabeled images;   identifying, by the computing system, a second training dataset having a plurality of labeled images, each of the plurality of labeled images associated with one of a plurality of classifications;   establishing, by the computing system, a second model using the second training dataset, the second model comprising:
 a convolution block having the first plurality of kernel parameters from the first model to generate a feature map using a corresponding labeled image of the plurality of labeled images; 
 an encoder having a second plurality of kernel parameters to generate a hash code for the corresponding labeled image based on the feature map; and 
 a classifier having a third plurality of kernel parameters to generate a classification for the corresponding labeled image based on the hash code; 
   determining, by the computing system, an error metric based on a comparison between the classification generated by the classifier and a corresponding classification indicated in the second training dataset; and   updating, by the computing system, the second model using the error metric to set at least one of the first plurality of kernel parameters in the convolution block, the second plurality of kernel parameters of the encoder, or the third plurality of kernel parameters of the classifier.   storing, by the computing system, at least one of a plurality of hash codes generated by the second model using the plurality of labeled images of the second training dataset.   
     
     
         2 . The method of  claim 1 , wherein the first model further comprises a generator comprising a fourth plurality of kernel parameters and a discriminator comprising the first plurality of kernel parameters. 
     
     
         3 . The method of  claim 1 , wherein establishing the second model further comprises transferring the first plurality of kernel parameters from a discriminator of the first model to the convolution block of the second model. 
     
     
         4 . The method of  claim 1 , wherein the encoder of the second model further comprises a threshold layer having at least a subset of the second plurality of kernel parameters to generate a first discrete value for the hash code when an input value satisfies a threshold and generate a second discrete value for the hash code when the input value does not satisfy the threshold. 
     
     
         5 . The method of  claim 1 , wherein each hash code of the plurality of hash codes has a set of values defining one or more features of the corresponding labeled image, the set of values of the hash code corresponding to at least one of a color, a texture, an object type, and semantic information. 
     
     
         6 . The method of  claim 1 , wherein the plurality of unlabeled images in the first training dataset are of a first domain and the plurality of labeled images in the second training dataset are of a second domain different from the first domain, each of the plurality of labeled images derived from a tissue sample via a histopathological image preparer; and
 wherein the plurality of labeled images in the second training dataset number less than the plurality of unlabeled images in the first training dataset.   
     
     
         7 . The method of  claim 1 , further comprising using, by the computing system, at least one of the plurality of hash codes generated by the second model to identify a matching labeled image of the plurality of labeled images. 
     
     
         8 . A method, comprising:
 identifying, by a computing system, a first biomedical image with which to find at least one of a plurality of second biomedical images;   applying, by the computing system, an image retrieval model to the first biomedical image, the image retrieval model comprising:
 a convolution block having a first plurality of kernel parameters to generate a feature map using the first biomedical image, the first plurality of kernel parameters transferred from a preliminary model; and 
 an encoder having a second plurality of kernel parameters to generate a first hash code for the first biomedical image based on the feature map; 
   selecting, by the computing system, from the plurality of second biomedical images corresponding to a plurality of second hash codes, a subset of second biomedical images using the first hash code for the first biomedical image; and   providing, by the computing system, the subset of second biomedical images identified using the first biomedical image.   
     
     
         9 . The method of  claim 8 , wherein selecting the subset of second biomedical images further comprises:
 generating a distance metric between the first hash code and a corresponding second hash code of the plurality of second hash codes for a corresponding second biomedical image of the plurality of second biomedical images;   determining that the distance metric between the first hash code and corresponding second hash code is within a threshold metric; and   including, into the subset of second biomedical images, the second biomedical image corresponding to the second hash code.   
     
     
         10 . The method of  claim 8 , wherein the encoder of the image retrieval model further comprises a threshold layer having at least a subset of the second plurality of kernel parameters to generate a first discrete value for the hash code when an input value to the threshold layer satisfies a threshold and generate a second discrete value for the hash code when the input value to the threshold layer does not satisfy the threshold. 
     
     
         11 . The method of  claim 8 , wherein each hash code of the plurality of hash codes has a set of values defining one or more features of the corresponding labeled image, the set of values of the hash code corresponding to at least one of a color, a texture, an object type, and semantic information. 
     
     
         12 . The method of  claim 8 , wherein the convolution block of the image retrieval model comprises the first plurality of kernel parameters transferred from the preliminary model, the preliminary model established using a training dataset having a plurality of unlabeled images different from a plurality of labeled images used to establish the image retrieval model. 
     
     
         13 . The method of  claim 8 , wherein the image retrieval model lacks a classifier used to update at least one of the first plurality of kernel parameters of the convolution block and the second plurality of kernel parameters of the encoder based on a comparison between a classification for a sample biomedical image generated by the classifier and a labeled classifier for the sample biomedical image as identified in a training dataset. 
     
     
         14 . The method of  claim 8 , wherein identifying the biomedical image further comprises receiving the first biomedical image derived from a tissue sample via a histopathological image preparer. 
     
     
         15 . A system, comprising:
 a computing system having one or more processors coupled with memory, configured to:
 identify a first biomedical image with which to find at least one of a plurality of second biomedical images; 
 apply an image retrieval model to the first biomedical image, the image retrieval model comprising:
 a convolution block having a first plurality of kernel parameters to generate a feature map using the first biomedical image, the first plurality of kernel parameters transferred from a preliminary model; and 
 an encoder having a second plurality of kernel parameters to generate a first hash code for the first biomedical image based on the feature map; 
 
 identify, from the plurality of second biomedical images corresponding to a plurality of second hash codes, a subset of second biomedical images using the first hash code for the first biomedical image; and 
 provide the subset of second biomedical images identified using the first biomedical image. 
   
     
     
         16 . The system of  claim 15 , wherein the computing system is further configured to:
 generate a distance metric between the first hash code and a corresponding second hash code of the plurality of second hash codes for a corresponding second biomedical image of the plurality of second biomedical images;   determine that the distance metric between the first hash code and corresponding second hash code is within a threshold metric; and   include, into the subset of second biomedical images, the second biomedical image corresponding to the second hash code.   
     
     
         17 . The system of  claim 15 , wherein the encoder of the image retrieval model further comprises a threshold layer having at least a subset of the second plurality of kernel parameters to generate a first discrete value for the hash code when an input value to the threshold layer satisfies a threshold and generate a second discrete value for the hash code when the input value to the threshold layer does not satisfy the threshold. 
     
     
         18 . The system of  claim 15 , wherein each hash code of the plurality of hash codes has a set of values defining one or more features of the corresponding labeled image, the set of values of the hash code corresponding to at least one of a color, a texture, an object type, and semantic information. 
     
     
         19 . The system of  claim 15 , wherein the convolution block of the image retrieval model comprises the first plurality of kernel parameters transferred from the preliminary model, the preliminary model established using a training dataset having a plurality of unlabeled images different from a plurality of labeled images used to establish the image retrieval model. 
     
     
         20 . The system of  claim 15 , wherein the computing system is further configured to receive the first biomedical image derived from a tissue sample via a histopathological image preparer.