Diagnosing skin conditions using machine-learned models

Abstract

A diagnosis system trains a set of machine-learned diagnosis models that are configured to receive an image of a patient and generate predictions on whether the patient has one or more health conditions. In one embodiment, the set of machine-learned models are trained to generate predictions for images that contain two or more underlying health conditions of the patient. In one instance, the symptoms for the two or more health conditions are shown as two or more overlapping skin abnormalities on the patient. By using the architectures of the set of diagnosis models described herein, the diagnosis system can generate more accurate predictions for images that contain overlapping symptoms for two or more health conditions compared to existing systems.

1 . A method of diagnosing overlapping skin abnormalities in an input image, the method comprising:
 receiving, from a client device, a request to diagnose skin abnormalities in an input image, the input image including overlapping skin abnormalities on skin of a patient;   accessing a set of machine-learned models from a database, each machine-learned model including a respective set of trained weights;   generating a respective prediction for each of two or more skin abnormalities in the input image by applying the set of machine-learned models to the input image, a prediction indicating a likelihood that a respective skin abnormality in the two or more skin abnormalities are shown in the input image;   generating diagnoses of the overlapping skin abnormalities from the predictions for the input image; and   providing, to the client device, the diagnoses.   
     
     
         2 . The method of  claim 1 , wherein the set of machine-learned models is an ensemble set of neural network models, and wherein generating the predictions for the two or more skin abnormalities further comprises:
 for each neural network model in the ensemble set, generating one or more predictions from the neural network model by applying the neural network model to the input image, and   combining the predictions for the ensemble set of neural network models to generate the predictions for the two or more skin abnormalities.   
     
     
         3 . The method of  claim 1 , wherein the set of machine-learned models includes a second machine-learned model and a third machine-learned model, and wherein generating the predictions for the two or more skin abnormalities further comprises:
 generating an image tensor by applying the second machine-learned model to the input image, the image tensor characterizing a plurality of spatial features in the input image,   extracting a plurality of components from the image tensor,   generating a respective tensor for each of the two or more skin abnormalities, and   generating the predictions for the two or more skin abnormalities by applying the third machined-learned model to the respective tensor for each of the two or more skin abnormalities.   
     
     
         4 . The method of  claim 3 , wherein the plurality of components are extracted from the image tensor by performing independent component analysis (ICA) on the image tensor. 
     
     
         5 . The method of  claim 3 , wherein the set of trained weights for the second machine-learned model and the third machine-learned model are jointly trained. 
     
     
         6 . The method of  claim 1 , wherein the set of machine-learned models includes a recurrent neural network model, and wherein generating the predictions for the two or more skin abnormalities further comprises:
 repeatedly applying the recurrent neural network model to the input image to generate the respective prediction for a first skin abnormality in the two or more skin abnormalities at a first time, and generate the respective prediction for a second skin abnormality in the two or more skin abnormalities at a second time subsequent the first time.   
     
     
         7 . The method of  claim 1 , wherein the set of machine-learned models includes a second machine-learned model, a third machine-learned model, and a fourth machine-learned model, and wherein generating the predictions for the two or more skin abnormalities further comprises:
 generating a prediction on whether the input image includes an amorphous skin abnormality or a localized abnormality, and   responsive to determining that the input image includes an amorphous skin abnormality, generating a prediction for the amorphous skin abnormality by applying the amorphous abnormality model to the input image,   responsive to determining that the input image includes a localized skin abnormality, generating a prediction for the localized skin abnormality by applying the localized abnormality model to the input image.   
     
     
         8 . The method of  claim 1 , wherein at least one of the set of machine-learned models are configured as a neural network architecture that includes a set of layers of nodes, each layer connected to a previous layer via a subset of weights. 
     
     
         9 . The method of  claim 1 , wherein the input image is at least one of a radiology image, a computerized tomography (CT) scan, a medical resonance imaging (MRI) scan, a X-ray image, an ultrasound or ultrasonography image, a tactile image, or a thermography images. 
     
     
         10 . The method of  claim 1 , wherein the input image is an image captured by a user of the client device, wherein the client device is a smartphone. 
     
     
         11 . A computer program product for diagnosing overlapping skin abnormalities in an input image, the computer program product comprising a computer-readable storage medium containing computer program code for:
 receiving, from a client device, a request to diagnose skin abnormalities in an input image, the input image including overlapping skin abnormalities on skin of a patient;   accessing a set of machine-learned models from a database, each machine-learned model including a respective set of trained weights;   generating a respective prediction for each of two or more skin abnormalities in the input image by applying the set of machine-learned models to the input image, a prediction indicating a likelihood that a respective skin abnormality in the two or more skin abnormalities are shown in the input image;   generating diagnoses of the overlapping skin abnormalities from the predictions for the input image; and   providing, to the client device, the diagnoses.   
     
     
         12 . The computer program product of  claim 11 , wherein the set of machine-learned models is an ensemble set of neural network models, and wherein generating the predictions for the two or more skin abnormalities further comprises:
 for each neural network model in the ensemble set, generating one or more predictions from the neural network model by applying the neural network model to the input image, and   combining the predictions for the ensemble set of neural network models to generate the predictions for the two or more skin abnormalities.   
     
     
         13 . The computer program product of  claim 11 , wherein the set of machine-learned models includes a second machine-learned model and a third machine-learned model, and wherein generating the predictions for the two or more skin abnormalities further comprises:
 generating an image tensor by applying the second machine-learned model to the input image, the image tensor characterizing a plurality of spatial features in the input image,   extracting a plurality of components from the image tensor,   generating a respective tensor for each of the two or more skin abnormalities, and   generating the predictions for the two or more skin abnormalities by applying the third machined-learned model to the respective tensor for each of the two or more skin abnormalities.   
     
     
         14 . The computer program product of  claim 13 , wherein the plurality of components are extracted from the image tensor by performing independent component analysis (ICA) on the image tensor. 
     
     
         15 . The computer program product of  claim 13 , wherein the set of trained weights for the second machine-learned model and the third machine-learned model are jointly trained. 
     
     
         16 . The computer program product of  claim 11 , wherein the set of machine-learned models includes a recurrent neural network model, and wherein generating the predictions for the two or more skin abnormalities further comprises:
 repeatedly applying the recurrent neural network model to the input image to generate the respective prediction for a first skin abnormality in the two or more skin abnormalities at a first time, and generate the respective prediction for a second skin abnormality in the two or more skin abnormalities at a second time subsequent the first time.   
     
     
         17 . The computer program product of  claim 11 , wherein the set of machine-learned models includes a second machine-learned model, a third machine-learned model, and a fourth machine-learned model, and wherein generating the predictions for the two or more skin abnormalities further comprises:
 generating a prediction on whether the input image includes an amorphous skin abnormality or a localized abnormality, and   responsive to determining that the input image includes an amorphous skin abnormality, generating a prediction for the amorphous skin abnormality by applying the amorphous abnormality model to the input image,   responsive to determining that the input image includes a localized skin abnormality, generating a prediction for the localized skin abnormality by applying the localized abnormality model to the input image.   
     
     
         18 . The computer program product of  claim 11 , wherein at least one of the set of machine-learned models are configured as a neural network architecture that includes a set of layers of nodes, each layer connected to a previous layer via a subset of weights. 
     
     
         19 . The computer program product of  claim 11 , wherein the input image is at least one of a radiology image, a computerized tomography (CT) scan, a medical resonance imaging (MRI) scan, a X-ray image, an ultrasound or ultrasonography image, a tactile image, or a thermography images. 
     
     
         20 . The computer program product of  claim 11 , wherein the input image is an image captured by a user of the client device, wherein the client device is a smartphone. 
     
     
         21 - 40 . (canceled)