Automatic generation of radiology reports from images and automatic rule out of images without findings

Abstract

A computer-implemented method for automatically generating a radiology report includes a computer receiving an input dataset comprising a plurality of multidimensional patient images and patient information and parsing the input dataset using learned models to determine a clinical domain and relevant image annotations. The computer populates an annotation table using the relevant image annotations and applies one or more domain-specific scriptable rules to populate a report template based on the annotation table. The computer may then generate a natural language radiology report based on the report template.

We claim: 
     
         1 . A computer-implemented method for automatically generating a radiology report, the method comprising:
 receiving, by a computer, an input dataset comprising a plurality of multidimensional patient images and patient information;   parsing, by the computer, the input dataset using learned models to determine a clinical domain and relevant image annotations;   populating, by the computer, an annotation table using the relevant image annotations;   applying, by the computer, one or more domain-specific scriptable rules to populate a report template based on the annotation table; and   generating, by the computer, a natural language radiology report based on the report template.   
     
     
         2 . The method of  claim 1 , further comprising:
 receiving, by the computer, an indication of a clinical study being performed on the input dataset,   wherein the natural language radiology report provides an explanation of a clinical finding relevant to the clinical study and one or more image features corresponding to the clinical finding.   
     
     
         3 . The method of  claim 2 , further comprising:
 presenting the natural language radiology report in an interactive graphical user interface which allows a user to retrieve images depicting the one or more image features via activation of one or more links embedded in the natural language radiology report.   
     
     
         4 . The method of  claim 1 , wherein the natural language radiology report comprises one or more recommendations for modifying a scanner acquisition protocol to acquire one or more additional patient images. 
     
     
         5 . The method of  claim 4 , further comprising:
 receiving, by the computer, an indication of a clinical study being performed on the plurality of multidimensional patient images;   detecting, by the computer, that target anatomy relevant to the clinical study is partially or completely out of the field of view of all of the plurality of multidimensional patient images,   wherein the one or more recommendations for modifying the scanner acquisition protocol comprise a recommended modification to patient positioning during imaging.   
     
     
         6 . The method of  claim 1 , wherein a rule-out process is applied to the plurality of multidimensional patient images prior parsing the plurality of multidimensional patient images using the learned models, the rule-out process comprising:
 receiving, by the computer, an indication of a clinical study being performed on the plurality of multidimensional patient images;   identifying, by the computer, a subset of the plurality of multidimensional patient images which are irrelevant to the clinical study; and   disregarding, by the computer, the subset of the plurality of multidimensional patient images from the input dataset or the input dataset as a whole.   
     
     
         7 . The method of  claim 1 , further comprising:
 performing an offline preparation process comprising:
 creating a clinical report template based on existing clinical reports and domain knowledge; 
 identifying one or more clinical report concepts and acceptable data ranges relevant to the clinical report concepts based on the existing clinical reports and the domain knowledge; 
 using the clinical report template, the one or more clinical report concepts, and the acceptable data ranges relevant to the clinical report concepts to create an annotation specification comprising one or more annotation tables and the one or more domain-specific scriptable rules; 
 using the annotation specification to create an annotation system; 
 applying the annotation system to one or more training images to yield one or more training image annotations; and 
 training one or more image parsing models based on the training images and the training image annotations. 
   
     
     
         8 . The method of  claim 7 , wherein the domain knowledge comprises one or more of clinical standards, clinical guidelines, or information provided in clinical consults. 
     
     
         9 . The method of  claim 7 , further comprising:
 using a basic report template provided in a Radiological Society of North America standardized format to create the clinical report template based on the existing clinical reports and domain knowledge.   
     
     
         10 . A computer-implemented method for automatically generating a radiology report, the method comprising:
 performing, by a computer, an offline training process comprising:
 determining a plurality of possible image annotations associated with a clinical domain; 
 determining a plurality of scriptable rules for populating a domain-specific clinical report template with information relevant to the plurality of possible image annotations; 
 training one or more domain specific models to parse image information and output one or more of the plurality of possible image annotations; and 
   performing, by the computer, an online report generation process comprising:
 receiving an input dataset comprising a plurality of multidimensional patient images and patient information; 
 deriving one or more relevant image annotations associated with the input dataset based on the plurality of possible image annotations associated with the clinical domain and the one or more domain specific models; 
 populating the domain-specific clinical report template using the one or more relevant image annotations and the plurality of scriptable rules; and 
 identifying one or more clinically relevant findings based on the populated domain-specific clinical report template. 
   
     
     
         11 . The method of  claim 10 , wherein the online report generation process further comprises:
 generating a natural language radiology report based on the one or more clinically relevant findings.   
     
     
         12 . The method of  claim 11 , wherein the online report generation process further comprises:
 presenting the natural language radiology report in an interactive graphical user interface which allows a user to retrieve images depicting image features relevant to the clinically relevant findings via activation of one or more links embedded in the natural language radiology report.   
     
     
         13 . The method of  claim 10 , wherein the online report generation process further comprises:
 identifying a change to an existing image acquisition protocol based on the one or more clinically relevant findings.   
     
     
         14 . The method of  claim 13 , wherein the online report generation process further comprises:
 automatically implementing the change to the existing image acquisition protocol on an image scanner to acquire one or more new images.   
     
     
         15 . The method of  claim 13 , wherein the online report generation process further comprises:
 displaying the change to the existing image acquisition protocol in a graphical user interface as a recommendation to a user.   
     
     
         16 . The method of  claim 15 , wherein the online report generation process further comprises:
 detecting, by the computer, that target anatomy relevant to the clinical domain is partially or completely out of the field of view of all of the plurality of multidimensional patient images,   wherein the change to the existing image acquisition protocol comprises a recommended modification to patient positioning during imaging.   
     
     
         17 . The method of  claim 13 , wherein the online report generation process further comprises:
 identifying, by the computer, a subset of the plurality of multidimensional patient images which are irrelevant to the clinical domain based on the one or more clinically relevant findings; and   disregarding, by the computer, the subset of the plurality of multidimensional patient images from the input dataset or the input dataset as a whole.   
     
     
         18 . A system for automatically generating a radiology report, the system comprising:
 a medical information database comprising one or more diagnostic multidimensional (e.g. 2D/3D/4D) image data and non-image patient metadata;   one or more processors configured to:
 communicate with the medical information database to retrieve a patient-specific input dataset; 
 parse the patient-specific input dataset using learned models to determine a clinical domain and relevant image annotations; 
 populate an annotation table using the relevant image annotations; 
 apply one or more domain-specific scriptable rules to populate a report template based on the annotation table; and 
 identify one or more clinically relevant findings based on the populated report template. 
   
     
     
         19 . The system of  claim 18 , wherein the one or more processors are further configured to:
 generate a natural language radiology report based on the report template.   
     
     
         20 . The system of  claim 19 , wherein the one or more processors are further configured to:
 present the natural language radiology report in an interactive graphical user interface which allows a user to retrieve images depicting image features relevant to the clinically relevant findings via activation of one or more links embedded in the natural language radiology report.