Methods for augmenting a surgical field with virtual guidance and tracking and adapting to deviation from a surgical plan

Abstract

One variation of a method includes: accessing a virtual patient model defining a target resected contour of a hard tissue of interest; after resection of the hard tissue of interest during a surgical operation, accessing an optical scan recorded by an optical sensor facing a surgical field occupied by a patient, detecting a set of features representing the patient in the optical scan, registering the virtual patient model to the hard tissue of interest in the surgical field based on the set of features, and detecting an actual resected contour of the hard tissue of interest in the optical scan; and calculating a spatial difference between the actual resected contour of the hard tissue of interest and the target resected contour of the hard tissue of interest represented in the virtual patient model registered to the hard tissue of interest in the surgical field.

I claim: 
     
         1 . A method for registering features of a patient in a surgical field comprising:
 accessing a virtual patient model representing a hard tissue of interest of the patient, the virtual patient model generated from a pre-operative scan of the hard tissue of interest of the patient;   during a first time period within a surgical operation and preceding incision of the patient proximal the hard tissue of interest:
 accessing a first sequence of optical scans captured by an optical sensor facing the surgical field occupied by the patient; and 
 detecting a constellation of visible soft tissue features on the patient proximal the hard tissue of interest; 
   generating a soft tissue field model representing:
 a first set of spatial relationships between visible soft tissue features in the constellation of visible soft tissue features; 
 a set of motional relationships between visible soft tissue features in the constellation of visible soft tissue features; and 
 a soft tissue gravity model characterizing deformation of the constellation of visible soft tissue features based on orientation of the hard tissue of interest; 
   during a second time period succeeding incision of the patient proximal the hard tissue of interest and prior to resection of the hard tissue of interest:
 accessing a second sequence of optical scans captured by the optical sensor; 
 detecting a first contour of the hard tissue of interest in the second sequence of optical scans; and 
 detecting the constellation of visible soft tissue features in the second sequence of optical scans; 
   registering a constellation of virtual hard tissue features defined in the virtual patient model to the first contour of the hard tissue of interest;   deriving a second set of spatial relationships between the constellation of virtual hard tissue features and the constellation of visible soft tissue features;   during a third time period succeeding the second time period and proximal resection of the hard tissue of interest:
 accessing a third sequence of optical scans captured by the optical sensor; and 
 detecting the constellation of visible soft tissue features in the third sequence of optical scans; and 
   aligning the virtual patient model to the constellation of visible soft tissue features detected in the third sequence of optical scans based on the soft tissue field model and the second set of spatial relationships.   
     
     
         2 . The method of  claim 1 , further comprising:
 detecting a second contour of the hard tissue of interest in the third sequence of optical scans; and   based on alignment of the virtual patient model to the constellation of visible soft tissue features detected in the third sequence of optical scans according to the soft tissue field model and the second set of spatial relationships, detecting a spatial difference between the constellation of virtual hard tissue features, defined in the virtual patient model, and the second contour of the hard tissue of interest detected in the third sequence of optical scans.   
     
     
         3 . The method of  claim 2 :
 wherein accessing the virtual patient model comprises accessing the virtual patient model comprising a virtual unresected femur of the patient representing the hard tissue of interest;   wherein detecting the first contour of the hard tissue of interest in the second sequence of optical scans comprises detecting an unresected contour of a femoral condyle of the patient, prior to resection, in the second sequence of optical scans;   wherein registering the constellation of virtual hard tissue features defined in the virtual patient model to the first contour of the hard tissue of interest comprises registering virtual unresected femoral condyle features defined in the virtual patient model to the unresected contour of the femoral condyle detected in the second sequence of optical scans;   wherein detecting the second contour of the hard tissue of interest in the third sequence of optical scans comprises detecting a resected contour of the femoral condyle in the third sequence of optical scans; and   wherein detecting the spatial difference comprises detecting the spatial difference between virtual unresected femoral condyle features defined in the virtual patient model and the resected contour of the femoral condyle detected in the third sequence of optical scans.   
     
     
         4 . The method of  claim 3 , further comprising:
 calculating a magnitude of resection of the femoral condyle based on the spatial difference;   calculating an orientation of resection of the femoral condyle based on the spatial difference;   characterizing a surface profile of the second contour detected in the third sequence of optical scans, the second contour comprising a resected contour of the femoral condyle; and   rendering the magnitude of resection, the orientation of resection, and the surface profile on a display present proximal the surgical field.   
     
     
         5 . The method of  claim 4 , wherein rendering the magnitude of resection, the orientation of resection, and the surface profile on the display comprises, during the third time period:
 detecting a position of an augmented reality headset, worn by a surgeon and comprising the display, proximal the surgical field;   estimating a perspective of the surgeon viewing the surgical field based on the position of the augmented reality headset;   generating an augmented reality frame comprising a projection of the virtual unresected femur of the patient from the perspective of the surgeon;   inserting the magnitude of resection, the orientation of resection, and the surface profile into the augmented reality frame; and   at the augmented reality headset, rendering the augmented reality frame.   
     
     
         6 . The method of  claim 3 :
 wherein accessing the virtual patient model comprises accessing the virtual patient model further comprising a virtual unresected tibia of the patient;   further comprising, during the second time period:
 detecting a second hard tissue of interest of the patient, prior to resection, in the first sequence of optical scans, the second hard tissue of interest comprising an unresected contour of a tibial plateau of the patient; and 
 registering the virtual unresected tibia defined in the virtual patient model to the second hard tissue of interest detected in the first sequence of optical scans; and 
   further comprising:
 deriving a third set of spatial relationships between the constellation of virtual hard tissue features and the virtual unresected tibia in the virtual patient model; 
 detecting a resected contour of the tibial plateau in the third sequence of optical scans; and 
 based on alignment of the virtual patient model to the constellation of visible soft tissue features detected in the third sequence of optical scans according to the soft tissue field model and the third set of spatial relationships, detecting a second spatial difference between virtual unresected tibial plateau features, defined in the virtual patient model, and the resected contour of the tibial plateau detected in the third sequence of optical scans. 
   
     
     
         7 . The method of  claim 1 :
 further comprising, deriving a mechanical axis of the hard tissue of interest based on detected movement of the constellation of visible soft tissue features within the first sequence of optical scans and the three-dimensional field model of the constellation of visible soft tissue features; and   wherein registering the constellation of virtual hard tissue features defined in the virtual patient model to the first contour of the hard tissue of interest comprises aligning the virtual patient model with the first contour of the hard tissue of interest based on the mechanical axis of the hard tissue of interest.   
     
     
         8 . The method of  claim 1 :
 further comprising, during the first time period:
 accessing an initial sequence of optical scans captured by the optical sensor; 
 detecting a head in the initial sequence of optical scans; 
 detecting a foot in the initial sequence of optical scans; 
 deriving an orientation of the patient relative to the optical sensor based on a location of the head and a location of the foot in the initial sequence of optical scans; 
 predicting a region of the surgical field occupied by the hard tissue of interest based on the orientation of the patient; 
 scanning the region in the surgical field depicted in the initial sequence of optical scans for a soft tissue proximal the hard tissue of interest; and 
 coarsely registering the virtual patient model to the soft tissue proximal the hard tissue of interest; and 
   wherein registering the constellation of virtual hard tissue features defined in the virtual patient model to the first contour of the hard tissue of interest comprises refining coarse registration of the virtual patient model to the hard tissue of interest based on alignment of virtual hard tissue features defined in the virtual patient model and the first contour of the hard tissue of interest detected in the second sequence of optical scans.   
     
     
         9 . The method of  claim 1 , further comprising, during a fourth time period succeeding the first time period and succeeding incision of the patient proximal the hard tissue of interest:
 accessing a fourth sequence of optical scans captured by the optical sensor;   detecting presence of a red surface in the third sequence of optical scans;   interpreting the red surface as an incision wound on the patient; and   confirming registration of the virtual patient model to the soft tissue proximal the hard tissue of interest in response to a location of the incision wound overlapping locations of virtual hard tissue features in the virtual patient model.   
     
     
         10 . The method of  claim 1 :
 wherein registering the constellation of virtual hard tissue features defined in the virtual patient model to the first contour of the hard tissue of interest comprises calculating a best-fit location of the virtual patient model, relative to the hard tissue of interest, that minimizes error between virtual hard tissue features defined in the virtual patient model and the first contour of the hard tissue of interest detected in the second sequence of optical scans; and   further comprising displacing virtual hard tissue features defined in the virtual patient model into alignment with the first contour of the hard tissue of interest detected in the second sequence of optical scans based on the soft tissue field model.   
     
     
         11 . The method of  claim 1 , further comprising:
 serving a prompt to a surgeon in the surgical field to manipulate a portion of the patient proximal the hard tissue of interest through a range of motion during the second time period; and   serving confirmation of registration of the virtual patient model to the hard tissue of interest to the surgeon.   
     
     
         12 . The method of  claim 1 :
 wherein accessing the first sequence of optical scans comprises:
 accessing a first sequence of color images from a fixed stereo camera arranged over and facing an operating table within the surgical field; and 
 transforming the first sequence of color images into a first set of three-dimensional color point clouds; 
   further comprising combining the first set of three-dimensional color point clouds into a composite three-dimensional color point cloud depicting hard tissue and soft tissue of the patient; and   wherein detecting the constellation of visible soft tissue features comprises selecting the constellation of visible soft tissue features from the composite three-dimensional color point cloud.   
     
     
         13 . The method of  claim 1 , further comprising, during the first time period:
 detecting a first orientation of the hard tissue of interest relative to gravity; and   deforming the constellation of visible soft tissue features according to the soft tissue gravity model based on the first orientation of the hard tissue of interest.   
     
     
         14 . The method of  claim 1 , further comprising:
 accessing a definition of a target resection of the hard tissue of interest of the patient;   calculating an actual resection of the hard tissue of interest of the patient based on the second contour of the hard tissue of interest detected in the third sequence of optical scans;   calculating a spatial difference between the actual resection of the hard tissue of interest and the target resection of the hard tissue of interest; and   rendering the spatial difference on a display present proximal the surgical field.   
     
     
         15 . The method of  claim 14 , further comprising:
 serving a prompt to a surgeon present proximal the surgical field to provide a reason for the spatial difference;   labeling the spatial difference as an intentional deviation from a surgical plan associated with the target resection of the hard tissue of interest based on the reason for the spatial difference presented by the surgeon; and   recording the spatial difference and the reason provided by the surgeon in a database and in association with the surgical operation.   
     
     
         16 . The method of  claim 1 , further comprising:
 accessing a definition of a target position of a surgical implant relative to the hard tissue of interest of the patient; and   during a fourth time period succeeding the third time period:
 accessing a fourth sequence of optical scans captured by the optical sensor; 
 detecting the constellation of visible soft tissue features in the fourth sequence of optical scans; 
 aligning the virtual patient model to the constellation of visible soft tissue features detected in the fourth sequence of optical scans based on the soft tissue field model and the second set of spatial relationships; 
 detecting the surgical implant in the fourth sequence of optical scans; 
 calculating an actual position of the surgical implant relative to virtual hard tissue features defined in the virtual patient model, aligned to the constellation of visible soft tissue features detected in the fourth sequence of optical scans according to the soft tissue field model and the second set of spatial relationships; and 
 calculating a spatial difference between the actual position of the surgical implant and the target position of the surgical implant defined relative to the hard tissue of interest of the patient. 
   
     
     
         17 . The method of  claim 16 , further comprising rendering the spatial difference on a display present proximal the surgical field. 
     
     
         18 . A method for registering features of a patient in a surgical field comprising:
 accessing a virtual patient model representing a hard tissue of interest of the patient, the virtual patient model generated from a pre-operative scan of the hard tissue of interest of the patient;   during a first time period within a surgical operation and preceding incision of the patient proximal the hard tissue of interest:
 accessing a first sequence of optical scans captured by an optical sensor facing the surgical field occupied by the patient; and 
 detecting a constellation of visible soft tissue features on the patient proximal the hard tissue of interest; 
   generating a soft tissue field model representing:
 a first set of spatial relationships between visible soft tissue features in the constellation of visible soft tissue features; and 
 a set of motional relationships between visible soft tissue features in the constellation of visible soft tissue features; 
   during a second time period succeeding incision of the patient proximal the hard tissue of interest and prior to resection of the hard tissue of interest:
 accessing a second sequence of optical scans captured by the optical sensor; 
 detecting a first contour of the hard tissue of interest in the second sequence of optical scans; and 
 detecting the constellation of visible soft tissue features in the second sequence of optical scans; 
   registering a constellation of virtual hard tissue features defined in the virtual patient model to the first contour of the hard tissue of interest; and   deriving a second set of spatial relationships between the constellation of virtual hard tissue features and the constellation of visible soft tissue features.   
     
     
         19 . The method of  claim 18 , further comprising:
 during a third time period succeeding the second time period and proximal resection of the hard tissue of interest, accessing a third sequence of optical scans captured by the optical sensor;   detecting the constellation of visible soft tissue features in the third sequence of optical scans;   aligning the virtual patient model to the constellation of visible soft tissue features detected in the third sequence of optical scans based on the soft tissue field model and the second set of spatial relationships;   detecting a second contour of the hard tissue of interest in the second sequence of optical scans; and   based on alignment of the virtual patient model to the constellation of visible soft tissue features detected in the third sequence of optical scans according to the soft tissue field model and the second set of spatial relationships, detecting a spatial difference between the constellation of virtual hard tissue features, defined in the virtual patient model, and the second contour of the hard tissue of interest detected in the third sequence of optical scans.   
     
     
         20 . The method of  claim 18 , further comprising:
 accessing a definition of a target position of a surgical implant relative to the hard tissue of interest of the patient;   during a third time period succeeding the second time period, accessing a third sequence of optical scans captured by the optical sensor;   detecting the constellation of visible soft tissue features in the third sequence of optical scans;   detecting the surgical implant in the third sequence of optical scans;   aligning the virtual patient model to the constellation of visible soft tissue features detected in the third sequence of optical scans based on the soft tissue field model and the second set of spatial relationships;   calculating an actual position of the surgical implant relative to virtual hard tissue features defined in the virtual patient model, aligned to the constellation of visible soft tissue features detected in the third sequence of optical scans according to the soft tissue field model and the second set of spatial relationships; and   calculating a spatial difference between the actual position of the surgical implant and the target position of the surgical implant defined relative to the hard tissue of interest of the patient.