Systems and user interfaces for enhancement of data utilized in machine-learning based medical image review

Abstract

Systems and techniques are disclosed for improvement of machine learning systems based on enhanced training data. An example method includes providing a visual concurrent display of a set of images of features, the features requiring classification by a reviewing user. The user interface is provided to enable the reviewing user to assign classifications to the images, the user interface being configured to create, read, update, and/or delete classifications. The user interface is responsive to the user, with the user response indicating at least two images with a single classification. The user interface is updated to represent the single classification.

1 - 20 . (canceled) 
     
     
         21 . A method executed by a hardware processor for facilitating classification of features in medical images comprising:
 providing, for access by a first user via an associated first user device, an interactive classification user interface displaying a medical image including a feature to be classified;   receiving, through the interactive classification user interface from the first user device, a first classification for the medical image;   receiving a second classification for the medical image;   in response to a discrepancy between the first classification for the medical image received from the first user device and the second classification for the medical image, determining an accuracy of each of the first user and a source associated with the second classification and assigning a final classification for the medical image based on the accuracy of each of the first user and the source; and   training a model with a machine learning system using the final classification for the medical image as training data.   
     
     
         22 . The method of  claim 21 , wherein the source associated with the second classification is a second user and wherein receiving the second classification includes receiving, through the interactive classification user interface, the second classification from a second user device associated with the second user. 
     
     
         23 . The method of  claim 21 , wherein the source associated with the second classification includes a machine-learned system. 
     
     
         24 . The method of  claim 21 , wherein determining the accuracy of each of the first user and the source includes determining, for each of the first user and the source, an accuracy in classifying a control image with a known classification. 
     
     
         25 . The method of  claim 24 , wherein assigning the final classification for the medical image includes assigning a first weight to the first classification based on the accuracy determined for the first user and assigning a second weight to the second classification based on the accuracy determined for the source. 
     
     
         26 . The method of  claim 25 , wherein determining the final classification for the medical image includes determining the final classification for the medical image based on a combination of the first weight assigned to the first classification and the second weight assigned to the second classification. 
     
     
         27 . The method of  claim 25 , where determining the final classification for the medical image includes determining the final classification for the medical image by discarding at least one of the first classification and the second classification based on the first weight assigned to the first classification and the second weight assigned to the second classification. 
     
     
         28 . The method of  claim 21 , further comprising, in response to determining the final classification based on the first classification, alerting the source of the second classification of the final classification. 
     
     
         29 . The method of  claim 21 , wherein the source associated with the second classification includes a machine-learned system and further comprising updating the machine-learned system in response to determining the final classification based on the first classification. 
     
     
         30 . The method of  claim 21 , wherein the model includes a feature detection model, the feature detection model being usable to classify other medical images as including a feature with a particular shape, size, malignancy risk, or clinical relevance. 
     
     
         31 . The method of  claim 21 , wherein the final classification is selected from a group consisting of a finding type, shape, border type, homogeneity, Breast Imaging Reporting and Data System (BIRADS) score. 
     
     
         32 . Non-transitory computer storage-media storing instructions that when executed by a system of one or more computers, cause the one or more computers to perform operations comprising:
 providing, for access by a first user via an associated first user device, an interactive classification user interface displaying a medical image including a feature to be classified;   receiving, through the interactive classification user interface from the first user device, a first classification for the medical image;   receiving a second classification for the medical image;   in response to a discrepancy between the first classification for the medical image received from the first user device and the second classification for the medical image, determining an accuracy of each of the first user and a source associated with the second classification and assigning a final classification for the medical image based on the accuracy of each of the first user and the source; and   training a model with a machine learning system using the final classification for the medical image as training data.   
     
     
         33 . The non-transitory computer storage-media of  claim 32 , wherein the source associated with the second classification is a second user and wherein receiving the second classification includes receiving, through the interactive classification user interface, the second classification from a second user device associated with the second user. 
     
     
         34 . The non-transitory computer storage-media of  claim 32 , wherein the source associated with the second classification includes a machine-learned system. 
     
     
         35 . The non-transitory computer storage-media of  claim 32 , wherein determining the accuracy of each of the first user and the source includes determining, for each of the first user and the source, an accuracy in classifying a control image with a known classification. 
     
     
         36 . The non-transitory computer storage-media of  claim 35 , wherein assigning the final classification for the medical image includes assigning a first weight to the first classification based on the accuracy determined for the first user and assigning a second weight to the second classification based on the accuracy determined for the source. 
     
     
         37 . The non-transitory computer storage-media of  claim 36 , where determining the final classification for the medical image includes determining the final classification for the medical image by discarding at least one of the first classification and the second classification based on the first weight assigned to the first classification and the second weight assigned to the second classification. 
     
     
         38 . A system comprising:
 one or more computers configured to:
 receive, from a first source, a first classification for a medical image including a feature to be classified; 
 receive, from a second source, a second classification for the medical image; 
 in response to a discrepancy between the first classification and the second classification for the medical image, determining an accuracy of each of the first source and the second source and assigning a final classification for the medical image based on the accuracy of each of the first source and the second source; and 
   training a model with a machine learning system using the final classification for the medical image as training data.   
     
     
         39 . The system of  claim 38 , wherein at least one of the first source and the second source is a machine-learned system. 
     
     
         40 . The system of  claim 38 , wherein assigning the final classification for the medical image includes assigning a first weight to the first classification based on the accuracy determined for the first source and assigning a second weight to the second classification based on the accuracy determined for the second source.