Systems and methods for generating histology image training datasets for machine learning models

Abstract

A system and method are provided for training and using a machine learning model to analyze hematoxylin and eosin (H&E) slide images, where the machine learning model is trained using a training data set comprising a plurality of unmarked H&E images and a plurality of marked H&E images, each marked H&E image being associated with one unmarked H&E image and each marked H&E image including a location of one or more molecules determined by analyzing a multiplex IHC image having at least two IHC stains, each IHC stain having a unique color and a unique target molecule. Predicted molecules and locations identified with the machine learning model result in an immunotherapy response class being assigned to the H&E slide image.

What is claimed: 
     
         1 . A computer-implemented method comprising:
 a. obtaining, at one or more processors, at least one H&E slide image associated with a biological specimen;   b. obtaining, at the one or more processors, one or more multiplex immunohistochemistry (IHC) images associated with the biological specimen, wherein each multiplex IHC image includes at least two IHC stains, where each IHC stain has a unique color and a unique target molecule;   c. for each multiplex IHC image, detecting, at the one or more processors, mixture colors comprised of more than one IHC stain and identifying the IHC stains that comprise each mixture color;   d. determining, at the one or more processors, the location of each IHC stain color and determining the location of the associated stained target molecules;   e. detecting, at the one or more processors, individual cell locations and determining which individual cells are lymphocytes;   f. for each H&E image and IHC image associated with the biological specimen, aligning and/or registering, at the one or more processors, images such that for each physical location in the biological specimen, all pixels associated with that physical location are aligned;   g. for each target molecule, marking, at the one or more processors, the location on the H&E image that corresponds to the locations of the target molecules stained on the IHC layers;   h. for each cell having a location that corresponds to the location of one or more IHC stains, calculating, at the one or more processors, the percentage of stained pixels overlapping the cell that is associated with each IHC stain to determine an IHC stain profile for each cell; and   i. storing marked and unmarked versions of the H&E image in a training data set.   
     
     
         2 . The computer-implemented method of  claim 1 , where the H&E image is captured from a tissue layer that is stained only with H&E. 
     
     
         3 . The computer-implemented method of  claim 1 , where the H&E image is captured from a tissue layer that is stained with H&E and at least one IHC stain. 
     
     
         4 . The computer-implemented method of  claim 1 , where the H&E image is a virtual H&E stain image generated based on cell and tissue structures visible in a brightfield image of a tissue layer. 
     
     
         5 . The computer-implemented method of  claim 1 , where determining the location of each IHC stain color includes setting an intensity threshold for each stain color and comparing the intensity of the stain color in each pixel to the intensity threshold for that stain color. 
     
     
         6 . The computer-implemented method of  claim 5 , further comprising generating an overlay for each IHC stain where each pixel having an intensity that exceeds the threshold for the IHC stain is annotated to indicate presence of the IHC stain in the pixel. 
     
     
         7 . The computer-implemented method of  claim 1 , wherein detecting cell locations is performed by a neural network. 
     
     
         8 . The computer-implemented method of  claim 1 , wherein identifying the IHC stains that comprise each mixture color is accomplished by deconvolving mixture colors within each image. 
     
     
         9 . The computer-implemented method of  claim 1 , further comprising assigning a tissue class to portions of the H&E image. 
     
     
         10 . The computer-implemented method of  claim 1 , further comprising associating an immunotherapy response score with the stored unmarked H&E image, based on clinical data associated with the biological specimen. 
     
     
         11 . The computer-implemented method of  claim 10 , wherein the immunotherapy response score is based on at least one of immunotherapy associated sequencing data, Immune Cell Infiltration, Immune Gene Expression Signatures, Multiplex PD-L1 and CD8 staining, and Multiplex macrophage IHC panels. 
     
     
         12 . The computer-implemented method of  claim 10 , wherein immunotherapy associated sequencing data includes at least one of tumor mutational burden (TMB), microsatellite Instability (MSI), and T Cell Clonality. 
     
     
         13 . The computer-implemented method of  claim 1 , repeating (a)-(i) for a plurality of biological specimens to generate the training data set. 
     
     
         14 . The computer-implemented method of  claim 1 , further comprising:
 receiving, at a histology image-based machine learning model, the unmarked H&E images from the training data set;   receiving, at the histology image-based machine learning model, data associated with each unmarked H&E image of the training data set; and   optimizing, using the unmarked H&E images from the training data set and the data associated with each unmarked H&E image, the histology image-based machine learning model to receive a subsequent unmarked H&E image and generate a report of predicted biomarker status associated with the unmarked H&E image or of a predicted immunotherapy response class associated with the unmarked H&E image.   
     
     
         15 . The computer-implemented method of  claim 14 ,
 wherein the biomarker status comprises at least one of a PD-L1 identification, immune cell type identification, immune cell location in a tumor region, a digital visual indication of locations of target molecules, cancer cell type identification, or a degree of mixing of cell types, tumor infiltration, and immune infiltration detected in the unmarked H&E image, or   wherein the immunotherapy response class is based on at least one of a number of predicted molecules or locations of the predicted molecules, and a comparison of a number of predicted molecules to a threshold for each molecule, and wherein the immunotherapy response class is a predicted immunotherapy response outcome.   
     
     
         16 . The computer-implemented method of  claim 14 , wherein the data associated with each unmarked H&E image includes, from the training data set, the corresponding marked H&E image for each unmarked H&E image, wherein the marked H&E image shows the location of IHC staining target molecules in one or more IHC images associated with the same biological specimen as the H&E image. 
     
     
         17 . The computer-implemented method of  claim 14 , where the data associated with each unmarked H&E image of the training data set includes an immunotherapy response score. 
     
     
         18 . The computer-implemented method of  claim 14 , where the histology image-based machine learning model is a neural network. 
     
     
         19 . A computing system for training a histology image-based machine learning model, comprising:
 one or more processors; and   one or more memories having stored thereon computer-executable instructions that, when executed by the one or more processors, cause the computing system to:   obtain at least one H&E slide image associated with a biological specimen;   obtain one or more multiplex immunohistochemistry (IHC) images associated with the biological specimen, wherein each multiplex IHC image includes at least two IHC stains, where each IHC stain has a unique color and a unique target molecule;   for each multiplex IHC image, detect mixture colors comprised of more than one IHC stain and identifying the IHC stains that comprise each mixture color;   determine the location of each IHC stain color and determining the location of the associated stained target molecules;   detect individual cell locations and determining which individual cells are lymphocytes;   for each H&E image and IHC image associated with the biological specimen, align and/or register images such that for each physical location in the biological specimen, all pixels associated with that physical location are aligned;   for each target molecule, mark, at the one or more processors, the location on the H&E image that corresponds to the locations of the target molecules stained on the IHC layers;   for each cell having a location that corresponds to the location of one or more IHC stains, calculate the percentage of stained pixels overlapping the cell that is associated with each IHC stain to determine an IHC stain profile for each cell; and   store marked and unmarked versions of the H&E image in a training data set.   
     
     
         20 . A computer-readable medium having stored thereon a set of computer-executable instructions that, when executed by one or more processors, cause a computer to:
 obtain at least one H&E slide image associated with a biological specimen;   obtain one or more multiplex immunohistochemistry (IHC) images associated with the biological specimen, wherein each multiplex IHC image includes at least two IHC stains, where each IHC stain has a unique color and a unique target molecule;   for each multiplex IHC image, detect mixture colors comprised of more than one IHC stain and identifying the IHC stains that comprise each mixture color;   determine the location of each IHC stain color and determining the location of the associated stained target molecules;   detect individual cell locations and determining which individual cells are lymphocytes;   for each H&E image and IHC image associated with the biological specimen, align and/or register images such that for each physical location in the biological specimen, all pixels associated with that physical location are aligned;   for each target molecule, mark, at the one or more processors, the location on the H&E image that corresponds to the locations of the target molecules stained on the IHC layers;   for each cell having a location that corresponds to the location of one or more IHC stains, calculate the percentage of stained pixels overlapping the cell that is associated with each IHC stain to determine an IHC stain profile for each cell; and   store marked and unmarked versions of the H&E image in a training data set.