Extended reality systems with three-dimensional visualizations of medical image scan slices

Abstract

A navigated surgery system includes at least one processor that is operative to obtain a 2D medical image slice of anatomical structure of a patient. The operations further obtain a 3D graphical model of anatomical structure. The operations determine a pose of a virtual cross-sectional plane extending through the 3D graphical model of the anatomical structure that corresponds to the anatomical structure of the 2D medical image slice. The operations control the XR headset to display the 2D medical image slice of the anatomical structure of the patient, display the 3D graphical model of the anatomical structure, and display a graphical object oriented with the pose relative to the 3D graphical model of the anatomical structure.

What is claimed is: 
     
         1 . A navigated surgery system comprising at least one processor operative to:
 obtain a first two-dimensional (2D) medical image slice of anatomical structure of a patient;   obtain a three-dimensional (3D) graphical model of anatomical structure;   determine a first pose of a first virtual cross-sectional plane extending through the 3D graphical model of the anatomical structure that corresponds to the anatomical structure of the first 2D medical image slice; and   control an extended reality (XR) headset to display the first 2D medical image slice of the anatomical structure of the patient, display the 3D graphical model of the anatomical structure, and display a first graphical object oriented with the first pose relative to the 3D graphical model of the anatomical structure.   
     
     
         2 . The navigated surgery system of  claim 1 , wherein the at least one processor is further operative to control the XR headset to display a graphical representation of a plane overlaid with the first pose on the 3D graphical model of the anatomical structure. 
     
     
         3 . The navigated surgery system of  claim 2 , wherein the graphical representation of the plane is provided to the XR headset for display as a shaded and/or colored box overlaid with the first pose on the 3D graphical model of the anatomical structure. 
     
     
         4 . The navigated surgery system of  claim 1 , wherein the at least one processor is further operative to:
 obtain a second 2D medical image slice of the anatomical structure of the patient, wherein the first 2D medical image slice is an angularly offset image slice of the anatomical structure of the patient relative to the second 2D medical image slice;   determine a second pose of a second virtual cross-sectional plane extending through the 3D graphical model of the anatomical structure that corresponds to the anatomical structure of the second 2D medical image slice; and   control the XR headset to display the second 2D medical image slice of the anatomical structure of the patient and to display a second graphical object oriented with the second pose relative to the 3D graphical model of the anatomical structure.   
     
     
         5 . The navigated surgery system of  claim 4 , wherein the first 2D medical image slice is an axial image slice of the anatomical structure of the patient and the second 2D medical image slice is a sagittal image slice of the anatomical structure of the patient. 
     
     
         6 . The navigated surgery system of  claim 4 , wherein the at least one processor is further operative to:
 control the XR headset to display a graphical representation of a first plane overlaid with the first pose on the 3D graphical model of the anatomical structure; and   control the XR headset to display a second graphical representation of a second plane overlaid with the second pose on the 3D graphical model of the anatomical structure.   
     
     
         7 . The navigated surgery system of  claim 6 , wherein the at least one processor is further operative to:
 control the XR headset to use a first color to render at least part of the first 2D medical image slice of the anatomical structure of the patient and to render at least part of the graphical representation of the first plane for display; and   control the XR headset to use a second color, which is different from the first color, to render at least part of the second 2D medical image slice of the anatomical structure of the patient and to render at least part of the graphical representation of the second plane for display.   
     
     
         8 . The navigated surgery system of  claim 4 , wherein the at least one processor is further operative to:
 obtain a third 2D medical image slice of the anatomical structure of the patient, wherein the third 2D medical image slice is an angularly offset image slice of the anatomical structure of the patient relative to the second and third 2D medical image slices;   determine a third pose of a third virtual cross-sectional plane extending through the 3D graphical model of the anatomical structure that corresponds to the anatomical structure of the third 2D medical image slice; and   control the XR headset to display the third 2D medical image slice of the anatomical structure of the patient and to display a third graphical object oriented with the third pose relative to the 3D graphical model of the anatomical structure.   
     
     
         9 . The navigated surgery system of  claim 8 , wherein:
 the first 2D medical image slice is an axial image slice of the anatomical structure of the patient;   the second 2D medical image slice is a sagittal image slice of the anatomical structure of the patient; and   the third 2D medical image slice is a coronal image slice of the anatomical structure of the patient.   
     
     
         10 . The navigated surgery system of  claim 1 , wherein the at least one processor is further operative to:
 obtain a tracked pose of a tip of a tool being tracked by a camera tracking system relative to the anatomical structure of the patient;   select the first 2D medical image slice from among a set of 2D medical image slices forming an imaged volume of the anatomical structure of the patient, based on the tracked pose of the tip of the tool relative to the anatomical structure of the patient.   
     
     
         11 . The navigated surgery system of  claim 1 , wherein the at least one processor is further operative to:
 responsive to a rotation command from a user, control the XR headset to display an angularly rotated view of the 3D graphical model of the anatomical structure while displaying the first graphical object oriented with the first pose.   
     
     
         12 . The navigated surgery system of  claim 1 , wherein the at least one processor is further operative to:
 obtain a tracked pose of the XR headset relative to the anatomical structure of the patient;   determine based on the tracked pose whether to flip orientation of a rendering of the first 2D medical image slice to be provided to the XR headset for display; and   select between two graphically distinct objects to be rendered as the first graphical object for display responsive to whether the determination is to flip orientation of the rendering of the first 2D medical image slice to be provided to the XR headset for display.   
     
     
         13 . The navigated surgery system of  claim 1 , wherein the at least one processor is further operative to:
 control the XR headset to display the 3D graphical object with the first pose and extending overlying a region of the 3D graphical model of the anatomical structure corresponding to where a surgical procedure is to be performed on the anatomical structure of the patient;   obtain a tracked pose of a tool or an implant being tracked by a camera tracking system relative to the anatomical structure of the patient;   control the XR headset to display a graphical representation of the tool or the implant overlaid with the tracked pose on the 3D graphical model of the anatomical structure while continuing to display the 3D graphical object.   
     
     
         14 . A navigated surgery system comprising at least one processor operative to:
 obtain a first two-dimensional (2D) medical image slice of a spinal anatomical structure of a patient;   obtain a three-dimensional (3D) graphical model of the spinal anatomical structure;   determine a first pose of a first virtual cross-sectional plane extending through the 3D graphical model of the spinal anatomical structure that corresponds to the spinal anatomical structure of the first 2D medical image slice; and   control an extended reality (XR) headset to display the first 2D medical image slice of the spinal anatomical structure of the patient, display the 3D graphical model of the spinal anatomical structure, and display a first graphical object oriented with the first pose relative to the 3D graphical model of the spinal anatomical structure   
       wherein the at least one processor is further operative to:
 obtain a second 2D medical image slice of the anatomical structure of the patient, wherein the first 2D medical image slice is an angularly offset image slice of the anatomical structure of the patient relative to the second 2D medical image slice; 
 determine a second pose of a second virtual cross-sectional plane extending through the 3D graphical model of the anatomical structure that corresponds to the anatomical structure of the second 2D medical image slice; and 
 control the XR headset to display the second 2D medical image slice of the anatomical structure of the patient and to display a second graphical object oriented with the second pose relative to the 3D graphical model of the anatomical stru. 
 
     
     
         15 . The navigated surgery system of  claim 14 , wherein the at least one processor is further operative to control the XR headset to display a graphical representation of a plane overlaid with the first pose on the 3D graphical model of the anatomical structure. 
     
     
         16 . The navigated surgery system of  claim 15 , wherein the graphical representation of the plane is provided to the XR headset for display as a shaded and/or colored box overlaid with the first pose on the 3D graphical model of the anatomical structure. 
     
     
         17 . The navigated surgery system of  claim 14 , wherein the first 2D medical image slice is an axial image slice of the anatomical structure of the patient and the second 2D medical image slice is a sagittal image slice of the anatomical structure of the patient. 
     
     
         18 . The navigated surgery system of  claim 14 , wherein the at least one processor is further operative to:
 control the XR headset to display a graphical representation of a first plane overlaid with the first pose on the 3D graphical model of the anatomical structure; and   control the XR headset to display a second graphical representation of a second plane overlaid with the second pose on the 3D graphical model of the anatomical structure.   
     
     
         19 . The navigated surgery system of  claim 18 , wherein the at least one processor is further operative to:
 control the XR headset to use a first color to render at least part of the first 2D medical image slice of the anatomical structure of the patient and to render at least part of the graphical representation of the first plane for display; and   control the XR headset to use a second color, which is different from the first color, to render at least part of the second 2D medical image slice of the anatomical structure of the patient and to render at least part of the graphical representation of the second plane for display.   
     
     
         20 . The navigated surgery system of  claim 14 , wherein the at least one processor is further operative to:
 obtain a third 2D medical image slice of the anatomical structure of the patient, wherein the third 2D medical image slice is an angularly offset image slice of the anatomical structure of the patient relative to the second and third 2D medical image slices;   determine a third pose of a third virtual cross-sectional plane extending through the 3D graphical model of the anatomical structure that corresponds to the anatomical structure of the third 2D medical image slice; and   control the XR headset to display the third 2D medical image slice of the anatomical structure of the patient and to display a third graphical object oriented with the third pose relative to the 3D graphical model of the anatomical structure.