Conditioning multi-modal patient data

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for jointly training an encoder neural network and a decoder neural network. In one aspect, a method comprises: updating current values of a set of encoder parameters and current values of a set of decoder parameters using gradients of a reconstruction loss function that measures an error in a reconstruction of multi-modal data from a training example, wherein: the reconstruction loss function comprises a plurality of scaling factors that each scale a respective term in the reconstruction loss function that measures an error in the reconstruction of a corresponding proper subset of feature dimensions of the multi-modal data from the training example.

What is claimed is: 
     
         1 . A method performed by one or more computers, the method comprising:
 receiving multi-modal data characterizing a target subject;   generating conditioning data for conditioning the multi-modal data characterizing the target subject based on a population of reference subjects, comprising:
 receiving, for each reference subject in the population of reference subjects, a feature representation of the reference subject corresponding to a reference modality and having a plurality of feature dimensions; and 
 generating the conditioning data based on the feature representations of the reference subjects; 
   applying the conditioning data to the multi-modal data characterizing the target subject; and   after applying the conditioning data to the multi-modal data characterizing the target subject, processing the multi-modal data characterizing the target subject using a machine learning model to generate a machine learning model output for the target subject.   
     
     
         2 . The method of  claim 1 , wherein generating the conditioning data based on the feature representations of the reference subjects comprises:
 determining, for each pair of feature dimensions comprising a first feature dimension and a second feature dimension from the plurality of feature dimensions, a respective correlation coefficient for the pair of feature dimensions that measures a correlation between:
 (i) a value of the first feature dimension in the feature representations of the reference subjects, and 
 (ii) a value of the second feature dimension in the feature representations of the reference subjects; and 
   generating the conditioning data based on the correlation coefficients.   
     
     
         3 . The method of  claim 2 , wherein for each reference subject in the population of reference subjects:
 the plurality of feature dimensions in the feature representation of the reference subject comprise a respective feature dimension corresponding to each protein in a predefined set of proteins; and   the value of each feature dimension corresponding to a protein defines an expression level of the protein in the reference subject.   
     
     
         4 . The method of  claim 2 , wherein for each reference subject in the population of reference subjects:
 the plurality of feature dimensions in the feature representation of the reference subject comprise a respective feature dimension corresponding to each gene in a predefined set of genes; and   the value of each feature dimension corresponding to a gene defines an expression level of the gene in the reference subject.   
     
     
         5 . The method of  claim 1 , further comprising receiving, for each reference subject in the population of reference subjects, a label that defines: (i) whether the reference subject has a particular medical condition, or (ii) whether the reference subject has responded to a treatment for a particular medical condition;
 wherein generating the conditioning data based on the feature representations of the reference subjects comprises:
 determining, for each feature dimension from the plurality of feature dimensions, a respective correlation coefficient that measures a correlation between:
 (i) a value of the feature dimension in the feature representations of the reference subjects, and 
 (ii) the labels of the reference subjects; and 
 
 generating the condition data based on the correlation coefficients. 
   
     
     
         6 . The method of  claim 1 , wherein for each reference subject in the population of reference subjects, receiving a feature representation of the reference subject corresponding to a reference modality comprises:
 receiving a pre-treatment feature representation of the reference subject captured before a medical treatment is applied to the reference subject; and   receiving a post-treatment feature representation of the reference subject captured after the medical treatment is applied to the reference subject.   
     
     
         7 . The method of  claim 6 , wherein generating the conditioning data based on the feature representations of the reference subjects comprises:
 generating, for each reference subject, a differential feature representation of the reference subject as a difference between: (i) the pre-treatment feature representation of the reference subject, and (ii) the post-treatment feature representation of the reference subject;   generating the conditioning data as a combination of the differential feature representations of the reference subjects.   
     
     
         8 . The method of  claim 7 , wherein generating the conditioning data as a combination of the differential feature representations of reference subjects comprises:
 generating the conditioning data as an average of the differential feature representations of the reference subjects.   
     
     
         9 . The method of  claim 6 , wherein the pre-treatment feature representation and the post-treatment feature representation of the reference subject are captured using functional magnetic resonance imaging (fMRI). 
     
     
         10 . The method of  claim 6 , wherein the pre-treatment feature representation and the post-treatment feature representation of the reference subject are captured using positron emission tomography (PET) imaging. 
     
     
         11 . The method of  claim 1 , wherein applying the conditioning data to the multi-modal data characterizing the target subject comprises:
 pointwise multiplying each of a plurality of feature dimensions of the multi-modal data by a corresponding dimension of the conditioning data.   
     
     
         12 . The method of  claim 1 , wherein the conditioning data is represented as a two-dimensional (2D) matrix of numerical values, and wherein applying the conditioning data to the multi-modal data characterizing the target subject comprises:
 matrix multiplying a plurality of feature dimensions of the multi-modal data by the 2D matrix of numerical values representing the conditioning data.   
     
     
         13 . The method of  claim 1 , wherein applying the conditioning data to the multi-modal data characterizing the target subject comprises:
 applying the conditioning data to a plurality of feature dimensions of the multi-modal data corresponding to a target modality,   wherein the target modality is a different modality than the reference modality used to generate the conditioning data.   
     
     
         14 . The method of  claim 1 , wherein the machine learning model comprises an encoder neural network, and wherein processing the multi-modal data characterizing the target subject using the machine learning model comprises:
 processing the multi-modal data characterizing the target subject using the encoder neural network to generate an embedding of the multi-modal data characterizing the target subject;   determining a respective classification score for each patient category in a set of patient categories based on the embedding of the multi-modal data characterizing the target subject; and   classifying the target subject as being included in a corresponding patient category from the set of patient categories based on the classification scores.   
     
     
         15 . The method of  claim 1 , wherein processing the multi-modal data characterizing the target subject using the machine learning model comprises:
 processing the multi-modal data characterizing the target subject using the machine learning model, in accordance with values of a plurality of machine learning model parameters, to generate a prediction characterizing the target subject.   
     
     
         16 . The method of  claim 15 , wherein the prediction characterizing the target subject comprises a prediction for whether the target subject has a particular medical condition. 
     
     
         17 . The method of  claim 1 , wherein the multi-modal data characterizing the target subject comprises a respective feature representation for each of a plurality of modalities. 
     
     
         18 . The method of  claim 18 , wherein each of the plurality of modalities corresponds to a respective sensor, and wherein the feature representation of each modality is based on data generated by the corresponding sensor. 
     
     
         19 . A system comprising:
 one or more computers; and   one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:   receiving multi-modal data characterizing a target subject;   generating conditioning data for conditioning the multi-modal data characterizing the target subject based on a population of reference subjects, comprising:
 receiving, for each reference subject in the population of reference subjects, a feature representation of the reference subject corresponding to a reference modality and having a plurality of feature dimensions; and 
 generating the conditioning data based on the feature representations of the reference subjects; 
   applying the conditioning data to the multi-modal data characterizing the target subject; and   after applying the conditioning data to the multi-modal data characterizing the target subject, processing the multi-modal data characterizing the target subject using a machine learning model to generate a machine learning model output for the target subject.   
     
     
         20 . One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
 receiving multi-modal data characterizing a target subject;   generating conditioning data for conditioning the multi-modal data characterizing the target subject based on a population of reference subjects, comprising:
 receiving, for each reference subject in the population of reference subjects, a feature representation of the reference subject corresponding to a reference modality and having a plurality of feature dimensions; and 
 generating the conditioning data based on the feature representations of the reference subjects; 
   applying the conditioning data to the multi-modal data characterizing the target subject; and   after applying the conditioning data to the multi-modal data characterizing the target subject, processing the multi-modal data characterizing the target subject using a machine learning model to generate a machine learning model output for the target subject.