Systems and Methods for Automatically Tagging Concepts to, and Generating Text Reports for, Medical Images Based On Machine Learning

Abstract

A system for assigning concepts to a medical image includes a visual feature module and a tagging module. The visual feature module is configured to obtain an image feature vector from the medical image. The tagging module is configured to apply a machine-learned algorithm to the image feature vector to assign a set of concepts to the image. The system may also include a text report generator that is configured to generate a written report describing the medical image based on the set of concepts assigned to the medical image.

1 . A method of assigning concepts to a medical image, the method comprising:
 obtaining an image feature vector from the medical image; and   applying a machine-learned algorithm to the image feature vector to assign a set of concepts to the medical image.   
     
     
         2 . The method of  claim 1 , wherein the image feature vector corresponds to an attentional representation of the medical image, and obtaining the image feature vector comprises:
 obtaining an image representation for the medical image;   obtaining a plurality of patch representations, each for a corresponding one of a plurality of patches of the medical image;   for each of the plurality of patches:
 generating an attention score based on the image representation and the patch representation; and 
 weighting image pixels of the patch based on the attention score; and 
   generating the attentional representation corresponding to the image feature vector based on the weighted image pixels.   
     
     
         3 . The method of  claim 1 , wherein applying a machine-learned algorithm to the image feature vector comprises:
 selecting one or more concepts to assign to the medical image based on a mapping function maintained in the machine-learned algorithm that maps concepts to medical images.   
     
     
         4 . The method of  claim 3 , wherein the mapping function maintained in the machine-learned algorithm is trained to map concepts to medical images based on:
 measures of relevance between medical images and concepts; and   measures of correlation among different concepts.   
     
     
         5 . The method of  claim 4 , wherein the concepts are included in an concept vector obtained by processing at least one concept ontology record with a long short-term memory (LSTM) recurrent neural network. 
     
     
         6 . The method of  claim 5 , wherein the LSTM is a tree-of-sequences LSTM. 
     
     
         7 . The method of  claim 1 , further comprising generating a written report describing the medical image based on the set of concepts assigned to the medical image. 
     
     
         8 . A system for assigning concepts to a medical image, the system comprising:
 a visual feature module configured to obtain an image feature vector from the medical image; and   a tagging module configured to apply a machine-learned algorithm to the image feature vector to assign a set of concepts to the medical image.   
     
     
         9 . The system of  claim 8 , wherein the image feature vector corresponds to an attentional representation of the medical image, and the visual feature module is configured to obtaining the image feature vector by being configured to:
 obtain an image representation for the medical image;   obtain a plurality of patch representations, each for a corresponding one of a plurality of patches of the medical image;   for each of the plurality of patches:
 generate an attention score based on the image representation and the patch representation; and 
 weight image pixels of the patch based on the attention score; and 
   generate the attentional representation corresponding to the image feature vector based on the weighted image pixels.   
     
     
         10 . The system of  claim 8 , wherein the machine-learned algorithm maintains a mapping function that maps concepts to medical images and the tagging module is configured to select one or more concepts to assign to the medical image based on the mapping function. 
     
     
         11 . The system of  claim 10 , wherein the mapping function maintained in the machine-learned algorithm is trained to map concepts to medical images based on:
 measures of relevance between medical images and concepts; and   measures of correlation among different concepts.   
     
     
         12 . The system of  claim 11 , wherein the concepts are included in an concept vector obtained by processing at least one concept ontology record with a long short-term memory (LSTM) recurrent neural network. 
     
     
         13 . The system of  claim 12 , wherein the LSTM is a tree-of-sequences LSTM. 
     
     
         14 . The system of  claim 8 , further comprising a text report generator configured to generate a written report describing the medical image based on the set of concepts assigned to the medical image. 
     
     
         15 . A machine learning apparatus for generating a map between medical image concepts and medical images, the apparatus comprising:
 a processor; and   a memory coupled to the processor; and   wherein the processor is configured to:
 generate representations of medical images in a form of image feature vectors; 
 generate representations of concepts describing medical images in a form of concept vectors; 
 process the image feature vectors and the concept vectors to obtain measures of relevance between images and concepts and measures of correlation among different concepts, and 
 associate each medical image represented in the image feature vector with one or more concepts represented in the concept vector based on the measures of relevance and the measures of correlation. 
   
     
     
         16 . The machine learning apparatus of  claim 15 , wherein the image feature vectors correspond to an attentional representation of medical images, and the processor obtains the image feature vector by being configured to:
 obtain an image representation for the medical image;   obtain a plurality of patch representations, each for a corresponding one of a plurality of patches of the medical image;   for each of the plurality of patches:
 generate an attention score based on the image representation and the patch representation; and 
 weight image pixels of the patch based on the attention score; and 
   generate the attentional representation corresponding to the image feature vector based on the weighted image pixels.   
     
     
         17 . The machine learning apparatus of  claim 15 , wherein the processor generates representations of concepts describing medical images in a form of concept vectors by processing at least one concept ontology record with a long short-term memory (LSTM) recurrent neural network. 
     
     
         18 . The machine learning apparatus of  claim 17 , wherein the LSTM is a tree-of-sequences LSTM. 
     
     
         19 . The machine learning apparatus of  claim 15 , wherein:
 a first set of image feature vectors and the concept vectors are processed to obtain the measures of relevance between images and concepts; and   a second set, different from the first set, of image feature vectors and the concept vectors are processed to obtain the measures of correlation among different concepts.