Interactive augmented reality system for laparoscopic and video assisted surgeries

Abstract

This disclosure describes an interactive augmented reality system for improving surgeon's view and context awareness during laparoscopic and video assisted surgeries. Instead of purely relying on computer vision algorithms for image registration between pre-operation (or intra-operation) images/models and later intra-operation scope images, the system can implement an interactive mechanism where surgeons may provide supervised information in initial calibration phase of the augmented reality function, thus achieving high accuracy in image registration. Besides the initialization phase before operation starts, interaction between surgeon and the system can also happens during the surgery. Specifically, patient tissue might move or deform during surgery, caused by for example cutting. The augmented reality system can re-calibrate during surgery when image registration accuracy deteriorates, by seeking additional supervised labeling from surgeons. The augmented reality system can improve surgeon's view during surgery, by utilizing surgeon's guidance sporadically to achieve high image registration accuracy.

What is claimed is: 
     
         1 . An apparatus comprising:
 a physical console configured for:
 receiving user-provided information that indicates a plurality of matching points between inner anatomical structures inside a body in reference images and corresponding inner anatomical structures inside the body in camera-captured images of a target field of view of a camera attached to a surgical instrument, wherein the camera-captured images are captured by the camera, wherein the matching points are determined after the reference images have been generated and after the camera-captured images have been captured by the camera; 
 performing image registration based on the received user-provided information that indicates the plurality of matching points between inner anatomical structures inside the body in reference images and corresponding inner anatomical structures inside the body in the camera-captured images; 
 overlaying, based on the image registration, augmented reality content on the camera-captured images, wherein the augmented reality content overlaid on the camera-captured images augments the view of the camera; 
 checking accuracy of the image registration based on the received user-provided information that indicates the plurality of matching points between inner anatomical structures inside the body in reference images and corresponding inner anatomical structures inside the body in the camera-captured images; 
 prompting a user to re-check the accuracy of the image registration; and 
 when the checked accuracy is below a pre-set threshold or when the user responds to the prompting by entering an input into the physical console indicating a redo, repeating the receiving step and the performing image registration step, 
   storage for recording the image registration, the storage included in the physical console or the storage separate from the physical console,   wherein the camera-captured images overlaid with the augmented reality content are displayed on a display, and   wherein the repeated receiving step and repeated performing image registration steps are based on a second instance of user-provided information that indicates a second plurality of matching points between inner anatomical structures inside the body in reference images and corresponding inner anatomical structures inside the body in camera-captures images of the camera attached to the surgical instrument.   
     
     
         2 . The apparatus of  claim 1 , wherein the surgical instrument comprises a laparoscope. 
     
     
         3 . The apparatus of  claim 1 , wherein the reference images are computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), or ultrasound images. 
     
     
         4 . The apparatus of  claim 1 , wherein the augmented reality content is based on a 3D model, which is based on the reference images. 
     
     
         5 . The apparatus of  claim 1 , wherein the augmented reality content includes visualized content that is hidden from the target field of view of the camera. 
     
     
         6 . The apparatus of  claim 1 , wherein the physical console is configured for:
 generating an alert of a risk situation.   
     
     
         7 . A method comprising:
 capturing, by a camera, images of a target field of view of the camera, the camera attached to a surgical instrument;   receiving user-provided information that indicates a plurality of matching points between inner anatomical structures inside a body in reference images and corresponding inner anatomical structures inside the body in the camera-captured images, wherein the matching points are determined after the reference images have been generated and after the camera-captured images have been captured by the camera;   performing image registration based on the received user-provided information that indicates the plurality of matching points between inner anatomical structures inside the body in reference images and corresponding inner anatomical structures inside the body in the camera-captured images;   overlaying, based on the image registration, augmented reality content on the camera-captured images, wherein the augmented reality content overlaid on the camera-captured images augments the view of the camera;   displaying the camera-captured images overlaid with the augmented reality content;   checking accuracy of the image registration based on the received user-provided information that indicates the plurality of matching points between inner anatomical structures inside the body in reference images and corresponding inner anatomical structures inside the body in the camera-captured images;   prompting a user to re-check the accuracy of the image registration; and   when the checked accuracy is below a pre-set threshold or when the user responds to the prompting by entering an input into the physical console indicating a redo, repeating the receiving step and the performing image registration step,   wherein the repeated receiving step and repeated performing image registration steps are based on a second instance of user-provided information that indicates a second plurality of matching points between inner anatomical structures inside the body in reference images and corresponding inner anatomical structures inside the body in camera-captures images of the camera attached to the surgical instrument.   
     
     
         8 . The method of  claim 7 , wherein the surgical instrument comprises a laparoscope. 
     
     
         9 . The method of  claim 7 , wherein the reference images are computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), or ultrasound images. 
     
     
         10 . The method of  claim 7 , wherein the augmented reality content is based on a 3D model, which is based on the reference images. 
     
     
         11 . The method of  claim 7 , wherein the augmented reality content includes visualized content that is hidden from the target field of view of the camera. 
     
     
         12 . The method of  claim 7 , the method comprising:
 generating an alert of a risk situation.   
     
     
         13 . A non-transitory machine-readable medium storing instructions, which when executed by one or more processors, cause the one or more processors to perform a method, the method comprising:
 receiving user-provided information that indicates a plurality of matching points between inner anatomical structures inside a body in reference images and corresponding inner anatomical structures inside the body in camera-captured images of a target field of view of a camera attached to a surgical instrument, wherein the camera-captured images are captured by the camera, wherein the matching points are determined after the reference images have been generated and after the camera-captured images have been captured by the camera;   performing image registration based on the received user-provided information that indicates the plurality of matching points between inner anatomical structures inside the body in reference images and corresponding inner anatomical structures inside the body in the camera-captured images;   overlaying, based on the image registration, augmented reality content on the camera-captured images, wherein the augmented reality content overlaid on the camera-captured images augments the view of the camera;   checking accuracy of the image registration based on the received user-provided information that indicates the plurality of matching points between inner anatomical structures inside the body in reference images and corresponding inner anatomical structures inside the body in the camera-captured images;   prompting a user to re-check the accuracy of the image registration; and   when the checked accuracy is below a pre-set threshold or when the user responds to the prompting by entering an input into the physical console indicating a redo, repeating the receiving step and the performing image registration step,   wherein the repeated receiving step and repeated performing image registration steps are based on a second instance of user-provided information that indicates a second plurality of matching points between inner anatomical structures inside the body in reference images and corresponding inner anatomical structures inside the body in camera-captures images of the camera attached to the surgical instrument, and   wherein the camera-captured images overlaid with the augmented reality content are displayed on a display.   
     
     
         14 . The non-transitory machine-readable medium of  claim 13 , wherein the reference images are computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), or ultrasound images. 
     
     
         15 . The non-transitory machine-readable medium of  claim 13 , wherein the augmented reality content is based on a 3D model, which is based on the reference images. 
     
     
         16 . The non-transitory machine-readable medium of  claim 13 , wherein the augmented reality content includes visualized content that is hidden from the target field of view of the camera. 
     
     
         17 . The non-transitory machine-readable medium of  claim 13 , the method comprising:
 generating an alert of a risk situation.   
     
     
         18 . An apparatus comprising:
 a physical console configured for:
 receiving user-provided information that indicates a plurality of matching points between inner anatomical structures inside a body in reference images and corresponding inner anatomical structures inside the body in camera-captured images of a target field of view of a camera attached to a surgical instrument, wherein the camera-captured images are captured by the camera, wherein the matching points are determined after the reference images have been generated and after the camera-captured images have been captured by the camera; 
 performing image registration based on the received user-provided information that indicates the plurality of matching points between inner anatomical structures inside the body in reference images and corresponding inner anatomical structures inside the body in the camera-captured images; 
 overlaying, based on the image registration, augmented reality content on the camera-captured images, wherein the augmented reality content overlaid on the camera-captured images augments the view of the camera; 
 tracking movements of the surgical instrument with the attached camera; 
 tracking movements of inner anatomical structures inside the body; 
 overlaying, based on the tracked movements of the surgical instrument with the attached camera and based on the tracked movements of inner anatomical structures inside the body, the augmented reality content overlaid on the camera-captured images to 
 follow movements of the camera and accommodate corresponding changes in the target field of view of the camera, and 
 follow movements of inner anatomical structures inside the body and accommodate corresponding changes in the target field of view of the camera; 
 during one or more of the tracking steps, checking accuracy of the image registration based on the received user-provided information that indicates the plurality of matching points between inner anatomical structures inside the body in reference images and corresponding inner anatomical structures inside the body in the camera-captured images; 
 when a user enters an input into the physical console indicating a redo, repeating the receiving step and the performing image registration step, and 
   storage for recording the image registration, the storage included in the physical console or the storage separate from the physical console,   wherein the camera-captured images overlaid with the augmented reality content are displayed on a display, and   wherein the repeated receiving step and repeated performing image registration steps are based on another instance of user-provided information that indicates another plurality of matching points between inner anatomical structures inside the body in reference images and corresponding inner anatomical structures inside the body in camera-captures images of the camera attached to the surgical instrument.