User input and interface design in augmented reality for use in surgical settings

Abstract

Various embodiments of an apparatus, methods, systems and computer program products described herein are directed to an Interaction Engine. According to various embodiments, the Interaction Engine generates within a unified three-dimensional (3D) coordinate space: (i) a virtual 3D model container; (ii) a virtual 3D medical model positioned according to a model pose within the virtual 3D model container; and (iii) a virtual 3D representation of at least a portion of at least one of a user's hands. The Interaction Engine renders an Augmented Reality (AR) display that includes concurrent display of the virtual 3D medical model container, the virtual 3D medical model and the virtual 3D representation of the user's hands. The Interaction Engine detects one or more physical gestures associated with the user and one or more types of virtual interacts associated with a detected physical gesture(s).

What is claimed is: 
     
       1. A computer-implemented method, comprising:
 generating, within a unified three-dimensional (3D) coordinate space: 
 (i) a virtual 3D medical model positioned according to a current model pose, the current model pose representing a position and orientation of the virtual 3D medical model in the unified 3D coordinate space; and 
 (ii) at least one a virtual 3D hand representation; 
 rendering, via an Augmented Reality (AR) headset device, an AR display that includes display of the virtual 3D medical model positioned according to the current model pose and the virtual 3D hand representation; 
 detecting a first physical gesture; 
 modifying the AR display, via the AR headset device, by rendering display of a type of movement of the virtual 3D hand representation based on the detected first physical gesture; 
 identifying selection of a clipping plane virtual interaction based on the type of movement of the virtual 3D hand representation; 
 modifying the AR display, via the AR headset device, by rendering a virtual clipping plane in the AR display concurrently displayed, at a first display position, with the virtual 3D medical model; 
 detecting a second physical gesture based on a first change of pose data of the AR headset device, the first change of pose data representing a change of a position and orientation of the AR headset device in the unified 3D coordinate space; 
 determining a second display position of the virtual clipping plane based on the first change of pose data of the AR headset device; 
 determining a first intersection between the virtual 3D clipping plane at the second display position and the rendered virtual 3D medical model at the current model pose; and 
 identifying one or more portions of non-rendered 3D medical model data for the virtual 3D medical model in the current model pose that map to the first intersection; and 
 modifying the AR display, via the AR headset device, by rendering the virtual clipping plane at the second display position concurrently with rendering the virtual 3D medical model portraying the identified one or more portions of 3D medical model data that map to the first intersection. 
 
     
     
       2. The computer-implemented method of  claim 1 , rendering, via an Augmented Reality (AR) headset device, an AR display that includes display of the virtual 3D medical mode positioned according to the model pose and a virtual 3D hand representation further comprises:
 rendering a virtual 3D model container; and 
 rendering the virtual 3D medical model positioned according to the model pose within a display of the rendered virtual 3D model container. 
 
     
     
       3. The computer-implemented method of  claim 2 , wherein identifying one or more portions of non-rendered 3D medical model data for the virtual 3D medical model in the current model pose that map to the first intersection comprises:
 identifying an overlap between one or more coordinates within the virtual 3D model container and one or more portions of the virtual 3D clipping plane bounded within the first intersection; 
 identifying one or more portions of non-rendered 3D medical model data that map to the identified one or more coordinates within the virtual 3D model container. 
 
     
     
       4. The computer-implemented method of  claim 3 , further comprising:
 prior to modifying the AR display by rendering the virtual clipping plane at the second display position:
 updating the virtual 3D medical model according to the identified one or more portions of 3D medical model data that map to the identified one or more coordinates within the virtual 3D model container. 
 
 
     
     
       5. The computer-implemented method of  claim 4 , further comprising:
 detecting a third physical gesture based on a subsequent second change of pose data of the AR headset device, the subsequent second change of pose data representing a change of a position and orientation of the AR headset device in the unified 3D coordinate space; 
 determining a third display position of the virtual clipping plane based on the subsequent change of pose data of the AR headset device; 
 determining a second intersection between the virtual 3D clipping plane at the third display position and the rendered updated virtual 3D medical model at the current model pose. 
 
     
     
       6. The computer-implemented method of  claim 5 , further comprising:
 identifying one or more portions of non-rendered 3D medical model data for the updated virtual 3D medical model in the current model pose that map to the second intersection; and 
 modifying the AR display, via the AR headset device, by rendering the virtual clipping plane at the third display position concurrently with rendering of a second updated virtual 3D medical model portraying the identified one or more portions of 3D medical model data that map to the second intersection. 
 
     
     
       7. The computer-implemented method of  claim 6 , further comprising:
 wherein rendering the virtual 3D medical model portraying the identified one or more portions of 3D medical model data that map to the first intersection comprises: rendering the virtual 3D medical model within the virtual 3D container; and 
 wherein rendering the second updated virtual 3D medical model portraying the identified one or more portions of 3D medical model data that map to the second intersection comprises: rendering the second updated virtual 3D medical model within the virtual 3D container. 
 
     
     
       8. A system comprising one or more processors, and a non-transitory computer-readable medium including one or more sequences of instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
 generating, within a unified three-dimensional (3D) coordinate space: 
 (i) a virtual 3D medical model positioned according to a current model pose, the current model pose representing a position and orientation of the virtual 3D medical model in the unified 3D coordinate space; and 
 (ii) at least one a virtual 3D hand representation; 
 rendering, via an Augmented Reality (AR) headset device, an AR display that includes display of the virtual 3D medical model positioned according to the current model pose and the virtual 3D hand representation; 
 detecting a first physical gesture; 
 modifying the AR display, via the AR headset device, by rendering display of a type of movement of the virtual 3D hand representation based on the detected first physical gesture; 
 identifying selection of a clipping plane virtual interaction based on the type of movement of the virtual 3D hand representation; 
 modifying the AR display, via the AR headset device, by rendering a virtual clipping plane in the AR display concurrently displayed, at a first display position, with the virtual 3D medical model; 
 detecting a second physical gesture based on a first change of pose data of the AR headset device, the first change of pose data representing a change of a position and orientation of the AR headset device in the unified 3D coordinate space; 
 determining a second display position of the virtual clipping plane based on the first change of pose data of the AR headset device; 
 determining a first intersection between the virtual 3D clipping plane at the second display position and the rendered virtual 3D medical model at the current model pose; and 
 identifying one or more portions of non-rendered 3D medical model data for the virtual 3D medical model in the current model pose that map to the first intersection; and 
 modifying the AR display, via the AR headset device, by rendering the virtual clipping plane at the second display position concurrently with rendering the virtual 3D medical model portraying the identified one or more portions of 3D medical model data that map to the first intersection. 
 
     
     
       9. The system of  claim 8 , rendering, via an Augmented Reality (AR) headset device, an AR display that includes display of the virtual 3D medical mode positioned according to the model pose and a virtual 3D hand representation further comprises:
 rendering a virtual 3D model container; and 
 rendering the virtual 3D medical model positioned according to the model pose within a display of the rendered virtual 3D model container. 
 
     
     
       10. The system of  claim 9 , wherein identifying one or more portions of non-rendered 3D medical model data for the virtual 3D medical model in the current model pose that map to the first intersection comprises:
 identifying an overlap between one or more coordinates within the virtual 3D model container and one or more portions of the virtual 3D clipping plane bounded within the first intersection; 
 identifying one or more portions of non-rendered 3D medical model data that map to the identified one or more coordinates within the virtual 3D model container. 
 
     
     
       11. The system of  claim 10 , further comprising:
 prior to modifying the AR display by rendering the virtual clipping plane at the second display position:
 updating the virtual 3D medical model according to the identified one or more portions of 3D medical model data that map to the identified one or more coordinates within the virtual 3D model container. 
 
 
     
     
       12. The system of  claim 11 , further comprising:
 detecting a third physical gesture based on a subsequent second change of pose data of the AR headset device, the subsequent second change of pose data representing a change of a position and orientation of the AR headset device in the unified 3D coordinate space; 
 determining a third display position of the virtual clipping plane based on the subsequent change of pose data of the AR headset device; 
 determining a second intersection between the virtual 3D clipping plane at the third display position and the rendered updated virtual 3D medical model at the current model pose. 
 
     
     
       13. The system of  claim 12 , further comprising:
 identifying one or more portions of non-rendered 3D medical model data for the updated virtual 3D medical model in the current model pose that map to the second intersection; and 
 modifying the AR display, via the AR headset device, by rendering the virtual clipping plane at the third display position concurrently with rendering of a second updated virtual 3D medical model portraying the identified one or more portions of 3D medical model data that map to the second intersection. 
 
     
     
       14. The system of  claim 13 , further comprising:
 wherein rendering the virtual 3D medical model portraying the identified one or more portions of 3D medical model data that map to the first intersection comprises: rendering the virtual 3D medical model within the virtual 3D container; and 
 wherein rendering the second updated virtual 3D medical model portraying the identified one or more portions of 3D medical model data that map to the second intersection comprises: rendering the second updated virtual 3D medical model within the virtual 3D container. 
 
     
     
       15. A computer program product comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein to be executed by one or more processors, the program code including instructions to:
 generating, within a unified three-dimensional (3D) coordinate space: 
 (i) a virtual 3D medical model positioned according to a current model pose, the current model pose representing a position and orientation of the virtual 3D medical model in the unified 3D coordinate space; and 
 (ii) at least one a virtual 3D hand representation; 
 rendering, via an Augmented Reality (AR) headset device, an AR display that includes display of the virtual 3D medical model positioned according to the current model pose and the virtual 3D hand representation; 
 detecting a first physical gesture; 
 modifying the AR display, via the AR headset device, by rendering display of a type of movement of the virtual 3D hand representation based on the detected first physical gesture; 
 identifying selection of a clipping plane virtual interaction based on the type of movement of the virtual 3D hand representation; 
 modifying the AR display, via the AR headset device, by rendering a virtual clipping plane in the AR display concurrently displayed, at a first display position, with the virtual 3D medical model; 
 detecting a second physical gesture based on a first change of pose data of the AR headset device, the first change of pose data representing a change of a position and orientation of the AR headset device in the unified 3D coordinate space; 
 determining a second display position of the virtual clipping plane based on the first change of pose data of the AR headset device; 
 determining a first intersection between the virtual 3D clipping plane at the second display position and the rendered virtual 3D medical model at the current model pose; and 
 identifying one or more portions of non-rendered 3D medical model data for the virtual 3D medical model in the current model pose that map to the first intersection; and 
 modifying the AR display, via the AR headset device, by rendering the virtual clipping plane at the second display position concurrently with rendering the virtual 3D medical model portraying the identified one or more portions of 3D medical model data that map to the first intersection. 
 
     
     
       16. The computer program product of  claim 15 , rendering, via an Augmented Reality (AR) headset device, an AR display that includes display of the virtual 3D medical mode positioned according to the model pose and a virtual 3D hand representation further comprises:
 rendering a virtual 3D model container; and 
 rendering the virtual 3D medical model positioned according to the model pose within a display of the rendered virtual 3D model container. 
 
     
     
       17. The computer program product of  claim 16 , wherein identifying one or more portions of non-rendered 3D medical model data for the virtual 3D medical model in the current model pose that map to the first intersection comprises:
 identifying an overlap between one or more coordinates within the virtual 3D model container and one or more portions of the virtual 3D clipping plane bounded within the first intersection; 
 identifying one or more portions of non-rendered 3D medical model data that map to the identified one or more coordinates within the virtual 3D model container. 
 
     
     
       18. The computer program product of  claim 17 , further comprising:
 prior to modifying the AR display by rendering the virtual clipping plane at the second display position:
 updating the virtual 3D medical model according to the identified one or more portions of 3D medical model data that map to the identified one or more coordinates within the virtual 3D model container. 
 
 
     
     
       19. The computer program product of  claim 18 , further comprising:
 detecting a third physical gesture based on a subsequent second change of pose data of the AR headset device, the subsequent second change of pose data representing a change of a position and orientation of the AR headset device in the unified 3D coordinate space; 
 determining a third display position of the virtual clipping plane based on the subsequent change of pose data of the AR headset device; 
 determining a second intersection between the virtual 3D clipping plane at the third display position and the rendered updated virtual 3D medical model at the current model pose; 
 identifying one or more portions of non-rendered 3D medical model data for the updated virtual 3D medical model in the current model pose that map to the second intersection; and 
 modifying the AR display, via the AR headset device, by rendering the virtual clipping plane at the third display position concurrently with rendering of a second updated virtual 3D medical model portraying the identified one or more portions of 3D medical model data that map to the second intersection. 
 
     
     
       20. A computer-implemented method, comprising:
 generating, within a unified three-dimensional (3D) coordinate space: 
 (i) a virtual 3D container; 
 (ii) a virtual 3D medical model positioned according to a current model pose within the virtual 3D container, the current model pose representing a position and orientation of the virtual 3D medical model in the unified 3D coordinate space; and 
 (iii) at least one a virtual 3D hand representation; 
 rendering, via an Augmented Reality (AR) headset device, an AR display that includes display of the virtual 3D container and virtual 3D medical model positioned according to the current model pose and the virtual 3D hand representation; 
 detecting a first physical gesture; 
 modifying the AR display, via the AR headset device, by rendering display of a type of movement of the virtual 3D hand representation based on the detected first physical gesture; 
 identifying selection of a cropping virtual interaction based on the type of movement of the virtual 3D hand representation; 
 modifying the AR display, via the AR headset device, by rendering a selectable shape in the AR display concurrently displayed on a first surface of the virtual 3D container, the first surface displayed at a first display position; 
 detecting a second physical gesture based on a first change of pose data of the AR headset device, the first change of pose data representing a change of a position and orientation of the AR headset device in the unified 3D coordinate space; 
 determining a movement of the selectable shape on the first surface of the virtual 3D container based on the first change of pose data of the AR headset device; 
 determining a second display position of the first surface of the virtual 3D container resulting from the movement of the selectable shape; 
 determining a first intersection between the first surface of the virtual 3D container at the second display position and the rendered virtual 3D medical model at the current model pose; and 
 identifying one or more portions of non-rendered 3D medical model data for the virtual 3D medical model in the current model pose that map to the first intersection; and 
 modifying the AR display, via the AR headset device, by rendering the first surface of the virtual 3D container at the second display position concurrently with rendering the virtual 3D medical model portraying the identified one or more portions of 3D medical model data that map to the first intersection.