Adversarial prediction of radiotherapy treatment plans

Abstract

Systems and methods are disclosed for generating radiotherapy treatment machine parameters based on projection images of a target anatomy. The systems and methods include operations including receiving a set of pairs of image data for each gantry angle of a radiotherapy treatment machine, wherein each pair of the set of pairs comprises a given projection image that represents a view of an anatomy of a subject from a given gantry angle and a given graphical aperture image of multi-leaf collimator (MLC) leaf positions at the given gantry angle based on the given projection image; training a generative adversarial network (GAN) model based on the set of pairs of image data for each gantry angle; and using the trained GAN model to predict an aperture image of MLC leaf positions for a desired gantry angle based on a projection image that represents a view of an anatomical region of interest.

What is claimed is: 
     
       1. A computer-implemented method comprising:
 receiving a set of pairs of image data for a plurality of gantry angles of a radiotherapy treatment machine, wherein a given pair of the set of pairs comprises a given projection image that represents a view of an anatomy of a subject from a given gantry angle and a given graphical aperture image of one or more multi-leaf collimator (MLC) leaf positions at the given gantry angle; and 
 training a neural network based on the set of pairs of image data for the plurality of gantry angles, wherein the trained neural network predicts an aperture image of one or more MLC leaf positions for a specified gantry angle based on a new projection image. 
 
     
     
       2. The method of  claim 1 , wherein the neural network comprises a generative adversarial network (GAN), wherein the GAN comprises a conditional adversarial network. 
     
     
       3. The method of  claim 1 , wherein the neural network comprises a cycle-consistent generative adversarial network (CycleGAN). 
     
     
       4. The method of  claim 1 , wherein the given gantry angle of a first pair of the set of pairs differs from the given gantry angle of a second pair of the set of pairs by a predetermined amount. 
     
     
       5. The method of  claim 1 , wherein:
 the neural network is configured to train a generative model using a discriminative model; 
 values applied by the generative model and the discriminative model are established using adversarial training between the discriminative model and the generative model; and 
 the generative model and the discriminative model comprise respective convolutional neural networks. 
 
     
     
       6. The method of  claim 5 , wherein:
 the adversarial training comprises: 
 training the generative model to generate a first synthetic graphical aperture image representation of the MLC leaf positions at a first gantry angle from a projection image that represents a view of a training subject anatomy from the first gantry angle; 
 training the discriminative model to classify the first synthetic graphical aperture image as a synthetic or a real training example graphical aperture image; and 
 an output of the generative model is used for training the discriminative model and an output of the discriminative model is used for training the generative model. 
 
     
     
       7. The method of  claim 6 , wherein the neural network is trained using a cycle-consistent generative adversarial network (CycleGAN) comprising the generative model and the discriminative model, wherein the generative model is a first generative model and the discriminative model is a first discriminative model, wherein the CycleGAN further comprises:
 a second generative model trained to: 
 process, from a given pair of the set of pairs, a given graphical aperture image representation of the MLC leaf positions at a given gantry angle as an input; 
 provide a synthetic projection image that represents a view of a training subject anatomy from the given gantry angle as an output; and 
 a second discriminative model trained to classify the synthetic projection image as a synthetic or a real projection image. 
 
     
     
       8. The method of  claim 7 , wherein the CycleGAN comprises a first portion to train the first generative model, the first portion trained to:
 obtain a set of training projection images representing different views of a patient anatomy from prior treatments that are paired with training graphical aperture images corresponding to each of the different views, each of the training graphical aperture images being aligned with a respective one of the training projection images; 
 transmit the set of training projection images to an input of the first generative model to output a first set of graphical aperture images; 
 receive the first set of graphical aperture images at an input of the first discriminative model to classify the first set of graphical aperture images as a synthetic or real training set of graphical aperture images; and 
 receive the first set of graphical aperture images at an input of the second generative model to generate a first set of cycle projection images, for calculating cycle-consistency losses. 
 
     
     
       9. The method of  claim 8 , wherein the CycleGAN comprises a second portion that is trained to:
 transmit the training graphical aperture images corresponding to each of the different views to the input of the second generative model to output a first set of synthetic projection images; 
 receive the first set of synthetic projection images at an input of the second discriminative model to classify the first set of synthetic projection images as synthetic or real training projection images; and 
 receive the first set of synthetic projection images at the input of the first generative model to generate a first set of cycle graphical aperture images for calculating cycle-consistency losses. 
 
     
     
       10. The method of  claim 9 , wherein:
 the cycle-consistency losses are generated based on a comparison of the first set of cycle projection images with the set of training projection images and a comparison of the first set of cycle graphical aperture images with the training graphical aperture images; 
 the first generative model is trained to minimize or reduce a first loss term that represents an expectation of difference between a plurality of synthetic graphical aperture images and respectively paired training graphical aperture images; and 
 the second generative model is trained to minimize or reduce a second loss term that represents an expectation of difference between a plurality of synthetic projection images and respectively paired training projection images. 
 
     
     
       11. The method of  claim 1 , further comprising:
 obtaining radiotherapy treatment machine parameter information representing one or more MLC leaf positions at gantry angles corresponding to a plurality of the views represented by one or more projection images of the set of pairs of image data and one or more radiotherapy beam intensities corresponding to each of the projection images; 
 generating training graphical aperture image representations based on the obtained radiotherapy treatment machine parameter information; and 
 aligning each of the generated training graphical aperture image representations with the corresponding one or more projection images. 
 
     
     
       12. The method of  claim 11 , wherein the training graphical aperture image representations and corresponding one or more projection images are two-dimensional images or three-dimensional images comprising stacks of two-dimensional projection image and graphical aperture image pairs corresponding to an entire treatment fraction. 
     
     
       13. A system comprising:
 one or more processors configured to perform operations comprising:
 receiving a set of pairs of image data for a plurality of gantry angles of a radiotherapy treatment machine, wherein a given pair of the set of pairs comprises a given projection image that represents a view of an anatomy of a subject from a given gantry angle and a given graphical aperture image of one or more multi-leaf collimator (MLC) leaf positions at the given gantry angle; and 
 training a neural network based on the set of pairs of image data for the plurality of gantry angles, wherein the trained neural network predicts an aperture image of one or more MLC leaf positions for a specified gantry angle based on a new projection image. 
 
 
     
     
       14. The system of  claim 13 , wherein the neural network comprises a generative adversarial network (GAN), wherein the GAN comprises a conditional adversarial network (eGAN) or a cycle-consistent generative adversarial network (CycleGAN). 
     
     
       15. The system of  claim 13 , wherein the given gantry angle of a first pair of the set of pairs differs from the given gantry angle of a second pair of the set of pairs by a predetermined amount. 
     
     
       16. The system of  claim 13 , wherein:
 the neural network is configured to train a generative model using a discriminative model; 
 values applied by the generative model and the discriminative model are established using adversarial training between the discriminative model and the generative model; and 
 the generative model and the discriminative model comprise respective convolutional neural networks. 
 
     
     
       17. The system of  claim 16 , wherein:
 the adversarial training comprises: 
 training the generative model to generate a first synthetic graphical aperture image representation of the MLC leaf positions at a first gantry angle from a projection image that represents a view of a training subject anatomy from the first gantry angle; 
 training the discriminative model to classify the first synthetic graphical aperture image as a synthetic or a real training example graphical aperture image; and 
 an output of the generative model is used for training the discriminative model and an output of the discriminative model is used for training the generative model. 
 
     
     
       18. A non-transitory computer-readable medium comprising non-transitory computer-readable instructions for performing operations comprising:
 receiving a set of pairs of image data for a plurality of gantry angles of a radiotherapy treatment machine, wherein a given pair of the set of pairs comprises a given projection image that represents a view of an anatomy of a subject from a given gantry angle and a given graphical aperture image of one or more multi-leaf collimator (MLC) leaf positions at the given gantry angle; and 
 training a neural network based on the set of pairs of image data for the plurality of gantry angles, wherein the trained neural network predicts an aperture image of one or more MLC leaf positions for a specified gantry angle based on a new projection image. 
 
     
     
       19. The non-transitory computer-readable medium of  claim 18 , wherein the neural network comprises a generative adversarial network (GAN), wherein the GAN comprises a conditional adversarial network (eGAN). 
     
     
       20. The non-transitory computer-readable medium of  claim 18 , wherein the neural network comprises a cycle-consistent generative adversarial network (CycleGAN). 
     
     
       21. The non-transitory computer-readable medium of  claim 18 , wherein the given gantry angle of a first pair of the set of pairs differs from the given gantry angle of a second pair of the set of pairs by a predetermined amount. 
     
     
       22. The non-transitory computer-readable medium of  claim 18 , wherein:
 the neural network is configured to train a generative model using a discriminative model; 
 values applied by the generative model and the discriminative model are established using adversarial training between the discriminative model and the generative model; and 
 the generative model and the discriminative model comprise respective convolutional neural networks. 
 
     
     
       23. The non-transitory computer-readable medium of  claim 22 , wherein:
 the adversarial training comprises: 
 training the generative model to generate a first synthetic graphical aperture image representation of the MLC leaf positions at a first gantry angle from a projection image that represents a view of a training subject anatomy from the first gantry angle; 
 training the discriminative model to classify the first synthetic graphical aperture image as a synthetic or a real training example graphical aperture image; and 
 an output of the generative model is used for training the discriminative model and an output of the discriminative model is used for training the generative model. 
 
     
     
       24. A computer-implemented method comprising:
 receiving a projection image that represents a view of an anatomical region of interest; 
 applying a neural network to the projection image to estimate an aperture image of one or more multi-leaf collimator (MLC) leaf positions for a specified gantry angle corresponding to the view of an anatomical region of interest, the neural network trained for a plurality of gantry angles of a radiotherapy treatment machine based on training data comprising image data for the plurality of gantry angles, wherein the image data comprises a given projection image that represents a view of an anatomy of a subject from a given gantry angle and a corresponding given graphical aperture image of one or more MLC leaf positions at the given gantry angle; and 
 generating a radiotherapy treatment plan based on the estimated aperture image of the MLC leaf positions. 
 
     
     
       25. The computer-implemented method of  claim 24 , further comprising generating one or more radiotherapy treatment machine parameters from the estimated aperture image. 
     
     
       26. The computer-implemented method of  claim 25 , further comprising computing a dose volume histogram or a three-dimensional dose distribution based on the generated one or more radiotherapy treatment machine parameters. 
     
     
       27. The computer-implemented method of  claim 25 , wherein the one or more radiotherapy treatment machine parameters include at least one of a gantry angle, an MLC jaw position, an MLC leaf position, or a radiation therapy beam intensity. 
     
     
       28. The computer-implemented method of  claim 25 , further comprising computing one or more dosimetric parameters based on the generated one or more radiotherapy treatment machine parameters. 
     
     
       29. The computer-implemented method of  claim 24 , wherein the given projection image in the image data is generated by ray tracing or Fourier reconstruction. 
     
     
       30. A system comprising:
 one or more processors configured to perform operations comprising: 
 receiving a projection image that represents a view of an anatomical region of interest; 
 applying a neural network to the projection image to estimate an aperture image of one or more multi-leaf collimator (MLC) leaf positions for a specified gantry angle corresponding to the view of an anatomical region of interest, the neural network trained for a plurality of gantry angles of a radiotherapy treatment machine based on training data comprising image data for the plurality of gantry angles, wherein the image data comprises a given projection image that represents a view of an anatomy of a subject from a given gantry angle and a corresponding given graphical aperture image of one or more MLC leaf positions at the given gantry angle; and 
 generating a radiotherapy treatment plan based on the estimated aperture image of the MLC leaf positions. 
 
     
     
       31. The system of  claim 30 , further comprising operations for generating one or more radiotherapy treatment machine parameters from the estimated aperture image. 
     
     
       32. The system of  claim 31 , further comprising operations for computing a dose volume histogram or a three-dimensional dose distribution based on the generated one or more radiotherapy treatment machine parameters. 
     
     
       33. The system of  claim 31 , wherein the one or more radiotherapy treatment machine parameters include at least one of a gantry angle, an MLC jaw position, an MLC leaf position, or a radiation therapy beam intensity. 
     
     
       34. The system of  claim 31 , further comprising computing one or more dosimetric parameters based on the generated one or more radiotherapy treatment machine parameters. 
     
     
       35. The system of  claim 30 , wherein the given projection image in the image data is generated by ray tracing or Fourier reconstruction.