Using augmented reality in surgical navigation

Abstract

A surgical navigation system may include a processor and a display. The processor may receive a patient image and sensor data captured by a sensor, receive a medical image, generate a hologram of the medical image, perform coregistration between the patient image and the hologram, superimpose the hologram on the patient image, and display the superimposed image. Coregistration may be performed manually via a user interaction, or automatically based on one or more fiducials in the medical image and sensor data related to the fiducials. The system may monitor a change in the environment and update the display correspondingly. For example, the system may monitor a movement of a body of the patient, monitor the size of an organ of the patient as the organ is being under operation, or a movement of the surgical instrument. The sensor may be an augmented reality (AR) sensor in an AR device.

What is claimed is: 
     
         1 . A system comprising:
 a processor;   a display; and   a computer readable non-transitory medium containing programming instructions that, when executed, will cause the processor to:
 receive a patient image comprising at least a body of a patient and sensor data captured by one or more augmented reality (AR) sensors; 
 receive a medical image; 
 generate a representation of the medical image; 
 perform coregistration between the patient image and the representation of the medical image to generate a transformation matrix; and 
 superimpose the representation of the medical image onto the patient image based on the transformation matrix to form a superimposed image; and 
 display the superimposed image on the display. 
   
     
     
         2 . The system of  claim 1 , wherein the programming instructions for performing the coregistration comprise programming instructions configured to:
 display the patient image on the display;   display the representation of the medical image on the display;   receive a user input to move the representation of the medical image to a location relative to the patient image on the display; and   generate the transformation matrix based on the relative location between the representation of the medical image and the patient image.   
     
     
         3 . The system of  claim 1 , wherein the programming instructions for performing the coregistration comprise programming instructions configured to generate the transformation matrix automatically by:
 extracting one or more features from the representation of the medical image;   generating volumetric data based on the sensor data; and   generating the transformation matrix based on the one or more features and the volumetric data.   
     
     
         4 . The system of  claim 3 , wherein:
 the one or more features include a fiducial; and   the sensor data comprises information about the fiducial.   
     
     
         5 . The system of  claim 4 , wherein the fiducial is a skin or an external surface of the patient image. 
     
     
         6 . The system of  claim 4 , wherein the fiducial is a deep structure of the body of the patient or a marker placed on the body of the patient. 
     
     
         7 . The system of  claim 4 , wherein the fiducial is an artery or septal divide between compartments of the body of the patient. 
     
     
         8 . The system of  claim 1 , wherein at least one of the one or more AR sensors includes a camera, a three-dimensional (3D) scanning device, or an ultrasound device. 
     
     
         9 . The system of  claim 1  further comprising additional programming instructions configured to determine a change of the body of the patient. 
     
     
         10 . The system of  claim 9 , wherein the additional programming instructions comprise programming instructions configured to:
 determine a movement of the body of the patient; and   if the movement of the body of the patient has exceeded a threshold, update the transformation matrix to generate an updated transformation matrix.   
     
     
         11 . The system of  claim 10 , wherein the programming instructions for updating the transformation matrix comprise programming instructions configured to:
 determine information about the movement of the body; and   update the transformation matrix based on the information about the movement of the body.   
     
     
         12 . The system of  claim 11 , wherein the information about the movement of the body comprises a position change of the body from a previous position. 
     
     
         13 . The system of  claim 10 , wherein the programming instructions for updating the transformation matrix comprise programming instructions configured to:
 receive an updated patient image; and   perform coregistration between the updated patient image and the representation of the medical image to generate the updated transformation matrix.   
     
     
         14 . The system of  claim 1 , wherein the representation of the medical image is a hologram. 
     
     
         15 . The system of  claim 11  further comprising additional programming instructions configured to:
 update the representation of the medical image based on the information about the movement of the body. 
 
     
     
         16 . The system of  claim 1 , wherein the body of the patient comprises at least one of a nerve, an artery, or an internal organ. 
     
     
         17 . The system of  claim 9 , wherein the additional programming instructions comprise programming instructions configured to:
 determine a change of the body of the patient in size; and   if the change of the size of the body of the patient has exceeded a threshold, update the transformation matrix to generate an updated transformation matrix.   
     
     
         18 . The system of  claim 9  further comprising additional programming instructions configured to:
 assess a function of a heart, a lung or an internal organ of the patient; or 
 assess a brain edema or blood loss. 
 
     
     
         19 . The system of  claim 1 , wherein the patient image includes a surgeon's hand or a surgical instrument in the surgeon's hand. 
     
     
         20 . The system of  claim 19  further comprising additional programming instructions configured to superimpose the surgeon's hand or the surgical instrument on the medical image. 
     
     
         21 . The system of  claim 20  further comprising additional programming instructions configured to:
 determine a change in a position or shape of the surgical instrument; and 
 superimpose the surgical instrument on the medical image based on the change in the position or the shape of the surgical instrument. 
 
     
     
         22 . The system of  claim 1 , wherein the display is a display of an AR device. 
     
     
         23 . The system of  claim 22 , wherein the display is configured to render a hologram. 
     
     
         24 . The system of  claim 22 , wherein the display is configured to display a 3D binocular vision. 
     
     
         25 . The system of  claim 22 , wherein the display is further configured to display an image of the patient image by a scaling factor, the scaling factor is equal or less than one. 
     
     
         26 . A method comprising:
 receiving a patient image comprising at least a body of a patient and sensor data captured by one or more augmented reality (AR) sensors;   receiving a medical image;   generating a representation of the medical image;   performing coregistration between the patient image and the representation of the medical image to generate a transformation matrix;   superimposing the representation of the medical image onto the patient image based on the transformation matrix to form a superimposed image; and   displaying the superimposed image on the display.   
     
     
         27 . The method of  claim 26 , wherein performing the coregistration comprises:
 extracting one or more features from the representation of the medical image;   generating volumetric data based on the sensor data; and   generating the transformation matrix based on the one or more features and the volumetric data.   
     
     
         28 . The method of  claim 26  further comprising:
 determining a movement of the body of the patient; and 
 if the movement of the body of the patient has exceeded a threshold, updating the transformation matrix to generate an updated transformation matrix. 
 
     
     
         29 . The method of  claim 26  further comprising:
 determining a movement of a surgical instrument in the patient image; and 
 superimposing the surgical instrument on the medical image based on the movement of the surgical instrument.