Image segmentation and object detection using fully convolutional neural network

Abstract

This disclosure relates to digital image segmentation, region of interest identification, and object recognition. This disclosure describes a method, a system, for image segmentation based on fully convolutional neural network including an expansion neural network and contraction neural network. The various convolutional and deconvolution layers of the neural networks are architected to include a coarse-to-fine residual learning module and learning paths, as well as a dense convolution module to extract auto context features and to facilitate fast, efficient, and accurate training of the neural networks capable of producing prediction masks of regions of interest. While the disclosed method and system are applicable for general image segmentation and object detection/identification, they are particularly suitable for organ, tissue, and lesion segmentation and detection in medical images.

What is claimed is: 
     
         1 . A system for performing segmentation of digital images, comprising:
 a communication interface circuitry;   a database;   a predictive model repository; and   a processing circuitry in communication with the database and the predictive model repository, the processing circuitry configured to:
 receive a set of training images labeled with a corresponding set of ground truth segmentation masks; 
 establish a fully convolutional neural network comprising a multi-layer contraction convolutional neural network and an expansion convolutional neural network connected in tandem; and 
 iteratively train the full convolution neural network in an end-to-end manner using the set of training images and the corresponding set of ground truth segmentation masks by configuring the processing circuitry to:
 down-sample a training image of the set of training images through the multi-layer contraction convolutional neural network to generate an intermediate feature map, wherein a resolution of the intermediate feature map is lower than a resolution of the training image, 
 up-sample the intermediate feature map through the multi-layer expansion convolutional neural network to generate a first feature map, 
 generate, based on the training image and the first feature map, a predictive segmentation mask for the training image, 
 generate an end loss based on a difference between the predictive segmentation mask and a ground truth segmentation mask corresponding to the training image, 
 back-propagate the end loss through the full convolutional neural network, and 
 minimize the end loss by adjusting a set of training parameters of the fully convolutional neural network using gradient descent. 
 
   
     
     
         2 . The system of  claim 1 , wherein the processing circuitry is further configured to:
 store the iteratively trained fully convolutional neural network with the set of training parameters in the predictive model repository;   receive an input image, wherein the input image comprises one of a test image or a unlabeled image; and   forward-propagate the input image through the iteratively trained fully convolutional neural network with the set of training parameters to generate an output segmentation mask.   
     
     
         3 . The system of  claim 1 , wherein when the processing circuitry is configured to generate, based on the training image and the first feature map, the predictive segmentation mask for the training image, the processing circuitry is configured to:
 implement a first auxiliary convolutional layer on the first feature map to generate a convoluted first feature map, the convoluted first feature map having a same resolution as the first feature map, the convoluted first feature map having a depth of one;   when the convoluted first feature map has a different resolution as the training image, adjust a resolution of the convoluted first feature map to have the same resolution as the training image; and   generate the predictive segmentation mask for the training image, based on the training image and the resolution-adjusted convoluted first feature map.   
     
     
         4 . The system of  claim 3 , wherein when the processing circuitry is configured to generate the predictive segmentation mask for the training image, based on the training image and the resolution-adjusted convoluted first feature map, the processing circuitry is configured to:
 perform a first sigmoid function on the resolution-adjusted convoluted first feature map to generate a first auxiliary prediction map; and   generate the predictive segmentation mask for the training image, based on the training image and the first auxiliary prediction map.   
     
     
         5 . The system of  claim 4 , wherein when the processing circuitry is configured to generate the predictive segmentation mask for the training image, based on the training image and the first auxiliary prediction map, the processing circuitry is configured to:
 add the training image and the first auxiliary prediction map to generate an auto-context prediction map, or concatenate the training image and the first auxiliary prediction map to generate the auto-context prediction map;   generate the predictive segmentation mask for the training image, based on the auto-context prediction map.   
     
     
         6 . The system of  claim 5 , wherein when the processing circuitry is configured to generate the predictive segmentation mask for the training image, based on the auto-context prediction map, the processing circuitry is configured to:
 perform a densely connected convolutional (DenseConv) operation on the auto-context prediction map to generate a DenseConv prediction map, the DenseConv operation including one or more convolutional layers;   perform a DenseConv auxiliary convolutional layer on the DenseConv prediction map to generate a convoluted DenseConv prediction map;   add the convoluted DenseConv prediction map and the added second feature map to generate the added DenseConv prediction map; and   generate, based on the added DenseConv prediction map, the predictive segmentation mask for the training image.   
     
     
         7 . The system of  claim 1 , wherein the processing circuitry is further configured to iteratively train the full convolution neural network in the end-to-end manner using the set of training images and the corresponding set of ground truth segmentation masks by configuring the processing circuitry to:
 up-sample the intermediate feature map through the multi-layer expansion convolutional neural network to generate a second feature map, wherein a resolution of the second feature map is larger than the resolution of the first feature map;   implement a first auxiliary convolutional layer on the first feature map to generate a convoluted first feature map, the convoluted first feature map having a same resolution as the first feature map, the convoluted first feature map having a depth of one;   implement a second auxiliary convolutional layer on the second feature map to generate a convoluted second feature map, the convoluted second feature map having a same resolution as the second feature map, the convoluted second feature map having a depth of one;   implement a first de-convolutional layer on the convoluted first feature map to generate a de-convoluted first feature map, the de-convoluted first feature map having a larger resolution than the first feature map;   add the de-convoluted first feature map and the convoluted second feature map to generate an added second feature map;   perform a second sigmoid function on the added second feature map to generate a second auxiliary prediction map; and   generate the predictive segmentation mask for the training image, based on the training image and the second auxiliary prediction map.   
     
     
         8 . The system of  claim 7 , wherein when the processing circuitry is configured to generate the predictive segmentation mask for the training image, based on the training image and the second auxiliary prediction map, the processing circuitry is configured to:
 add the training image and the second auxiliary prediction map to generate an auto-context prediction map, or concatenate the training image and the second auxiliary prediction map to generate an auto-context prediction map; and   generate the predictive segmentation mask for the training image, based on the auto-context prediction map.   
     
     
         9 . A method for image segmentation, comprising:
 receiving, by a computer comprising a memory storing instructions and a processor in communication with the memory, a set of training images labeled with a corresponding set of ground truth segmentation masks;   establishing, by the computer, a fully convolutional neural network comprising a multi-layer contraction convolutional neural network and an expansion convolutional neural network connected in tandem; and   iteratively training, by the computer, the full convolution neural network in an end-to-end manner using the set of training images and the corresponding set of ground truth segmentation masks by:
 down-sampling a training image of the set of training images through the multi-layer contraction convolutional neural network to generate an intermediate feature map, wherein a resolution of the intermediate feature map is lower than a resolution of the training image; 
 up-sampling the intermediate feature map through the multi-layer expansion convolutional neural network to generate a first feature map; 
 generating, based on the training image and the first feature map, a predictive segmentation mask for the training image; 
 generating an end loss based on a difference between the predictive segmentation mask and a ground truth segmentation mask corresponding to the training image; 
 back-propagating the end loss through the full convolutional neural network; and 
 minimizing the end loss by adjusting a set of training parameters of the fully convolutional neural network using gradient descent. 
   
     
     
         10 . The method of  claim 9 , further comprising:
 storing, by the computer, the iteratively trained fully convolutional neural network with the set of training parameters in a predictive model repository;   receiving, by the computer, an input image, wherein the input image comprises one of a test image or a unlabeled image; and   forward-propagating, by the computer, the input image through the iteratively trained fully convolutional neural network with the set of training parameters to generate an output segmentation mask.   
     
     
         11 . The method of  claim 9 , wherein the generating, based on the training image and the first feature map, the predictive segmentation mask for the training image comprises:
 implementing, by the computer, a first auxiliary convolutional layer on the first feature map to generate a convoluted first feature map, the convoluted first feature map having a same resolution as the first feature map, the convoluted first feature map having a depth of one;   when the convoluted first feature map has a different resolution as the training image, adjusting, by the computer, a resolution of the convoluted first feature map to have the same resolution as the training image; and   generating, by the computer, the predictive segmentation mask for the training image, based on the training image and the resolution-adjusted convoluted first feature map.   
     
     
         12 . The method of  claim 11 , wherein the generating the predictive segmentation mask for the training image, based on the training image and the resolution-adjusted convoluted first feature map, comprises:
 performing a first sigmoid function on the resolution-adjusted convoluted first feature map to generate a first auxiliary prediction map; and   generating the predictive segmentation mask for the training image, based on the training image and the first auxiliary prediction map.   
     
     
         13 . The method of  claim 12 , wherein the generating the predictive segmentation mask for the training image, based on the training image and the first auxiliary prediction map comprises:
 adding the training image and the first auxiliary prediction map to generate an auto-context prediction map, or concatenating the training image and the first auxiliary prediction map to generate the auto-context prediction map;   generating the predictive segmentation mask for the training image, based on the auto-context prediction map.   
     
     
         14 . The method of  claim 13 , wherein the generating the predictive segmentation mask for the training image, based on the auto-context prediction map, comprises:
 performing a densely connected convolutional (DenseConv) operation on the auto-context prediction map to generate a DenseConv prediction map, the DenseConv operation including one or more convolutional layers;   performing a DenseConv auxiliary convolutional layer on the DenseConv prediction map to generate a convoluted DenseConv prediction map;   adding the convoluted DenseConv prediction map and the added second feature map to generate the added DenseConv prediction map; and   generating, based on the added DenseConv prediction map, the predictive segmentation mask for the training image.   
     
     
         15 . The method of  claim 9 , wherein the iteratively training the full convolution neural network in the end-to-end manner using the set of training images and the corresponding set of ground truth segmentation masks further comprises:
 up-sampling the intermediate feature map through the multi-layer expansion convolutional neural network to generate a second feature map, wherein a resolution of the second feature map is larger than the resolution of the first feature map;   implementing a first auxiliary convolutional layer on the first feature map to generate a convoluted first feature map, the convoluted first feature map having a same resolution as the first feature map, the convoluted first feature map having a depth of one;   implementing a second auxiliary convolutional layer on the second feature map to generate a convoluted second feature map, the convoluted second feature map having a same resolution as the second feature map, the convoluted second feature map having a depth of one;   implementing a first de-convolutional layer on the convoluted first feature map to generate a de-convoluted first feature map, the de-convoluted first feature map having a larger resolution than the first feature map;   adding the de-convoluted first feature map and the convoluted second feature map to generate an added second feature map;   performing a second sigmoid function on the added second feature map to generate a second auxiliary prediction map; and   generating the predictive segmentation mask for the training image, based on the training image and the second auxiliary prediction map.   
     
     
         16 . The method of  claim 15 , wherein the generating the predictive segmentation mask for the training image, based on the training image and the second auxiliary prediction map comprises:
 adding the training image and the second auxiliary prediction map to generate an auto-context prediction map, or concatenating the training image and the second auxiliary prediction map to generate an auto-context prediction map; and   generating the predictive segmentation mask for the training image, based on the auto-context prediction map.   
     
     
         17 . A system for performing segmentation of digital images, comprising:
 a communication interface circuitry;   a database;   a predictive model repository; and   a processing circuitry in communication with the database and the predictive model repository, the processing circuitry configured to:
 receive a set of training images labeled with a corresponding set of ground truth segmentation masks; 
 establish a segmentation network comprising a first fully convolutional neural network, a second fully convolutional neural network, and an evaluation network, wherein each of the first and second fully convolutional neural networks comprises a multi-layer contraction convolutional neural network and an expansion convolutional neural network connected in tandem, and the evaluation network is in communication with the first and second fully convolutional neural networks; and 
 iteratively train the segmentation network in an end-to-end manner using the set of training images and the corresponding set of ground truth segmentation masks by configuring the processing circuitry to:
 generate a first predictive segmentation mask for a training image of the set of training images, by the first fully convolutional neural network based on the training image, 
 generate a second predictive segmentation mask for the training image, by the second fully convolutional neural network based on the training image, 
 generate a final predictive segmentation mask for the training image, by the evaluation network based on the first predictive segmentation mask and the second predictive segmentation mask, 
 generate an end loss based on a difference between the final predictive segmentation mask and a ground truth segmentation mask corresponding to the training image, 
 back-propagate the end loss through the segmentation network, and 
 minimize the end loss by adjusting a set of training parameters of the segmentation network using gradient descent. 
 
   
     
     
         18 . The system of  claim 17 , wherein the processing circuitry is further configured to:
 store the iteratively trained segmentation network with the set of training parameters in the predictive model repository;   receive an input image, wherein the input image comprises one of a test image or a unlabeled image; and   forward-propagate the input image through the iteratively trained segmentation network with the set of training parameters to generate an output segmentation mask.   
     
     
         19 . The system of  claim 17 , wherein when the processing circuitry is configured to generate the final predictive segmentation mask for the training image, by the evaluation network based on the first predictive segmentation mask and the second predictive segmentation mask, the processing circuitry is configured to:
 generate a final predictive value for each pixel of the final predictive segmentation mask, by the evaluation network based on values of corresponding pixels of the first predictive segmentation mask and the second predictive segmentation mask.   
     
     
         20 . The system of  claim 17 , wherein when the processing circuitry is configured to generate the final predictive segmentation mask for the training image, by the evaluation network based on the first predictive segmentation mask and the second predictive segmentation mask, the processing circuitry is configured to:
 add, by the evaluation network, the first predictive segmentation mask and the second predictive segmentation mask to generate the final predictive segmentation mask.