Procedure guidance and training apparatus, methods and systems

Abstract

Procedure training apparatus, methods and systems which in one embodiment allow surgical trainees to receive expert clinical guidance during live surgery or to practice surgical skills on anatomical models in a realistic manner with an augmented reality headset which is in audio and visual communication with a remotely located trainer having a similar AR headset. A unique feature of the system includes providing live or static imaging of the surgical site to the AR headsets of the trainee and trainer wherein the imaging is provided from medical or other devices (e.g., ultrasound) through a device(s) media hub. The images are displayed in the field of view of the surgeon so that they may see their hands, the patient (or model) and the imaging simultaneously. Controls allow the imaging on the respective AR headsets to be switched on command typically by the trainer which allows the trainer to deliver the trainer's surgical site image (e.g., on their own model) to the AR headset of the trainee to illustrate correct procedure, for example. Once the trainee views the trainer's imaging of correct procedure, the imaging source may be switched back to the trainee's surgical site so that they may continue the procedure. Instruction or live clinical support during surgical procedures allow procedures to be recorded and recordings segregated by machine learning model and class, allowing for semi-supervised and/or unsupervised learning. After machine learning models are created, automated guidance and/or alerts may be given during the surgical procedure.

What is claimed is: 
     
         1 . A system for providing communication between a trainee and a trainer of a procedure, the system comprising:
 a headset wearable by the trainee and being configured to communicate with a trainer computing device, the headset having a mounted camera oriented to capture a Field of View (FOV) corresponding to the viewable area in front of the trainee, the headset also having a viewing pane that is at least partially transparent such that the viewing pane may selectively 1) display augmented reality images which partially obstruct vision of objects physically beyond the viewing pane, and 2) enter a transparent state displaying no virtual images and no augmented reality images, wherein the headset and the computing device coordinate to transmit augmented reality images, wherein the objects physically beyond the viewing pane comprise a live procedure adjacent to the headset or the computing device; and   a trainee pointer in the headset and a trainer pointer associated with the computing device, the trainee pointer and trainer pointer enabling the trainee and trainer to indicate to the other an Area of Interest (AOI) within the FOV relative to the live procedure, wherein the headset is configured to display in the viewing pane a boundary of the AOI, the headset and computing device being configured to communicate with one another whether or not the FOV of the headset is oriented toward the AOI.   
     
     
         2 . The system of  claim 1  wherein the trainee pointer or trainer pointer is a laser pointer. 
     
     
         3 . The system of  claim 1  wherein the trainee pointer and the trainer pointer are selectively activated by the trainee and trainer, respectively. 
     
     
         4 . The system of  claim 1  wherein the trainer can direct the headset to display to the trainee the AOI. 
     
     
         5 . The system of  claim 1  wherein the trainer can request, through the computing device, a confirmation that the AOI is within the FOV of the headset. 
     
     
         6 . The system of  claim 1  wherein the trainee pointer and trainer pointer allow the trainee and trainer to highlight the AOI with a geometric shape. 
     
     
         7 . The system of  claim 1  wherein the trainee pointer and trainer pointer are adjustable to indicate an AOI as small as a single point or a geometric shape as large as desirable within the FOV. 
     
     
         8 . The system of  claim 1  wherein the headset is configured to calculate a direction and magnitude of a discrepancy between the FOV of the first headset and the AOI. 
     
     
         9 . The system of  claim 8  wherein the headset is configured to display an adjustment direction corresponding to the discrepancy. 
     
     
         10 . The system of  claim 8  wherein the headset is configured to display an adjustment magnitude corresponding to the discrepancy. 
     
     
         11 . The system of  claim 1  wherein an object that is part of the live procedure contains an identifying mark recognizable by the headset, wherein the identifying mark is recognizable by the headset as an AOI. 
     
     
         12 . The system of  claim 11  wherein the object is a glove worn by the trainer or trainee. 
     
     
         13 . The system of  claim 11  wherein the object is held by the trainer or the trainee. 
     
     
         14 . The system of  claim 11  wherein the mark includes an instruction to the headset to adjust the FOV. 
     
     
         15 . A method for operating a headset to orient a Field of View (FOV) of the headset toward a desired object, the method comprising:
 providing a headset having a see-through-optic display and having a viewing pane configured to display augmented reality images that correspond with a live procedure the headset, wherein the headset is coordinated to exchange information including views with a computing device, the headset having a pointer that can selectively indicate an Area of Interest (AOI) of an object within the FOV;   receiving an AOI from the headset via the pointer;   displaying the AOI using augmented reality images on at least one of the headset or the computing device;   comparing a FOV of the headset with the AOI to determine whether or not the AOI is within the FOV of the headset; and   notifying whether or not the AOI is within the FOV of the headset.   
     
     
         16 . The method of  claim 15  wherein the pointer is a laser pointer, and wherein the AOI received from the headset via the pointer is displayed on the computing device. 
     
     
         17 . The method of  claim 15 , further comprising determining that the AOI is not within the FOV, and issuing a notification that the AOI is not within the FOV. 
     
     
         18 . The method of  claim 17 , further comprising determining a direction between the AOI and the FOV. 
     
     
         19 . The method of  claim 18 , further comprising determining a magnitude of a distance between the AOI and the FOV. 
     
     
         20 . The method of  claim 15  wherein the live procedure includes a visible object having a code, the code pertaining to desired viewing parameters of the object, the method further comprising the headset reading the code and executing the viewing parameters.