Using augmented reality in surgical navigation

Abstract

A surgical navigation system may include a processor and a display. The processor may receive a patient image and sensor data captured by a sensor, receive a medical image, generate a hologram of the medical image, perform coregistration between the patient image and the hologram, superimpose the hologram on the patient image, and display the superimposed image. Coregistration may be performed manually via a user interaction, or automatically based on one or more fiducials in the medical image and sensor data related to the fiducials. The system may monitor a change in the environment and update the display correspondingly. For example, the system may monitor a movement of a body of the patient, monitor the size of an organ of the patient as the organ is being under operation, or a movement of the surgical instrument. The sensor may be an augmented reality (AR) sensor in an AR device.

What is claimed is: 
     
       1. A system comprising:
 a processor; 
 a display; and 
 a computer readable non-transitory medium containing programming instructions that, when executed, will cause the processor to:
 receive a first patient image showing at least a portion of a body of a patient and sensor data captured by one or more augmented reality (AR) sensors; 
 receive a medical image showing at least one anatomical structure that is not detectable by the one or more AR sensors; 
 generate a representation of the medical image; 
 perform coregistration between the first patient image and the representation of the medical image to generate a transformation matrix; 
 superimpose the representation of the medical image onto the first patient image based on the transformation matrix to form a superimposed image; 
 display the superimposed image on the display; 
 receive a second patient image showing the at least a portion of the body of the patient and sensor data captured by the one or more AR sensors; 
 determine a change in the at least one anatomical structure based on a change between the first patient image and the second patient image, wherein the change is caused by cutting or drilling within the body of the patient during a surgical intervention; and 
 update the superimposed image on the display based on detecting the change in the at least one anatomical structure. 
 
 
     
     
       2. The system of  claim 1 , wherein the programming instructions for performing the coregistration comprise programming instructions configured to:
 display the patient image on the display; 
 display the representation of the medical image on the display; 
 receive a user input to move the representation of the medical image to a location relative to the patient image on the display; and 
 generate the transformation matrix based on the relative location between the representation of the medical image and the patient image. 
 
     
     
       3. The system of  claim 1 , wherein the programming instructions for performing the coregistration comprise programming instructions configured to generate the transformation matrix automatically by:
 extracting one or more features from the representation of the medical image; 
 generating volumetric data based on the sensor data; and 
 generating the transformation matrix based on the one or more features and the volumetric data. 
 
     
     
       4. The system of  claim 3 , wherein the programming instructions that are configured to generate the transformation matrix automatically by extracting the one or more features from the representation of the medical image comprise instructions that are configured to extract one or more fiducials from the representation of the medical image, and wherein the sensor data comprises information about the fiducial. 
     
     
       5. The system of  claim 4 , wherein the instructions that are configured to extract one or more fiducials from the representation of the medical image comprise instructions that are configured to extract skin or an external surface of the patient image from the representation of the medical image. 
     
     
       6. The system of  claim 4 , wherein the instructions that are configured to extract one or more fiducials from the representation of the medical image comprise instructions that are configured to extract a deep structure of the body of the patient or a marker placed on the body of the patient from the representation of the medical image. 
     
     
       7. The system of  claim 4 , wherein the instructions that are configured to extract one or more fiducials from the representation of the medical image comprise instructions that are configured to extract an artery or septal divide between compartments of the body of the patient from the representation of the medical image. 
     
     
       8. The system of  claim 1 , wherein at least one of the one or more AR sensors includes a camera, a three-dimensional (3D) scanning device, or an ultrasound device. 
     
     
       9. The system of  claim 1 , further comprising additional programming instructions configured to determine a change of the body of the patient. 
     
     
       10. The system of  claim 9 , wherein the additional programming instructions comprise programming instructions configured to:
 determine a movement of the body of the patient; and 
 update the transformation matrix to generate an updated transformation matrix if the movement of the body of the patient has exceeded a threshold. 
 
     
     
       11. The system of  claim 10 , wherein the programming instructions for updating the transformation matrix comprise programming instructions configured to: determine information about the movement of the body; and update the transformation matrix based on the information about the movement of the body. 
     
     
       12. The system of  claim 11 , wherein the information about the movement of the body comprises a position change of the body from a previous position. 
     
     
       13. The system of  claim 11 , further comprising additional programming instructions configured to: update the representation of the medical image based on the information about the movement of the body. 
     
     
       14. The system of  claim 10 , wherein the programming instructions for updating the transformation matrix comprise programming instructions configured to:
 receive an updated patient image; and 
 perform coregistration between the updated patient image and the representation of the medical image to generate the updated transformation matrix. 
 
     
     
       15. The system of  claim 9 , wherein the additional programming instructions comprise programming instructions configured to:
 determine a change of the body of the patient in size; and 
 update the transformation matrix to generate an updated transformation matrix if the change of the size of the body of the patient has exceeded a threshold. 
 
     
     
       16. The system of  claim 9 , further comprising additional programming instructions configured to: assess a function of a heart, a lung or an internal organ of the patient; or assess a brain edema or blood loss. 
     
     
       17. The system of  claim 1 , wherein the representation of the medical image is a hologram. 
     
     
       18. The system of  claim 1 , wherein the body of the patient comprises at least one of a nerve, an artery, or an internal organ. 
     
     
       19. The system of  claim 1 , wherein the patient image includes a surgeon's hand or a surgical instrument in the surgeon's hand. 
     
     
       20. The system of  claim 1 , wherein the display is a display of an AR device. 
     
     
       21. The system of  claim 20 , wherein the display is configured to render a hologram. 
     
     
       22. The system of  claim 20 , wherein the display is configured to display a 3D binocular vision. 
     
     
       23. The system of  claim 20 , wherein the display is further configured to display an image of the patient image by a scaling factor, the scaling factor is equal or less than one. 
     
     
       24. The system of  claim 1 , wherein the at least one anatomical structure comprises a tumor, wherein the surgical intervention is a tumor resection, and wherein the change in the at least one anatomical structure comprises removal of at least a portion of the tumor. 
     
     
       25. The system of  claim 1 , wherein the at least one anatomical structure comprises a nerve, wherein the surgical intervention comprises drilling of a temporal bone, and wherein the change in the at least one anatomical structure of the medical image comprises movement of the nerve. 
     
     
       26. A method comprising:
 receiving a patient image showing at least a portion of a body of a patient and sensor data captured by one or more augmented reality (AR) sensors; 
 receiving a medical image; 
 generating a representation of the medical image; 
 performing coregistration between the patient image and the representation of the medical image to generate a transformation matrix; 
 superimposing the representation of the medical image onto the patient image based on the transformation matrix to form a superimposed image; 
 displaying the superimposed image on the display; 
 receiving a second patient image showing the at least a portion of the body of the patient and sensor data captured by the one or more AR sensors; 
 determining a change in the at least one anatomical structure based on a change between the first patient image and the second patient image, wherein the change is caused by cutting or drilling within the body of the patient during a surgical intervention; and 
 updating the superimposed image on the display based on detecting the change in the at least one anatomical structure. 
 
     
     
       27. The method of  claim 26 , wherein performing the coregistration comprises:
 extracting one or more features from the representation of the medical image; 
 generating volumetric data based on the sensor data; and 
 generating the transformation matrix based on the one or more features and the volumetric data. 
 
     
     
       28. The method of  claim 26 , further comprising:
 determining a movement of the body of the patient; and 
 updating the transformation matrix to generate an updated transformation matrix if the movement of the body of the patient has exceeded a threshold. 
 
     
     
       29. The method of  claim 26 , further comprising:
 determining a movement of a surgical instrument in the patient image; and 
 superimposing the surgical instrument on the medical image based on the movement of the surgical instrument.