Systems and methods for arbitrary viewpoint robotic manipulation and robotic surgical assistance

Abstract

Systems and methods for arbitrary viewpoint robotic manipulation and robotic surgical assistance are disclosed. According to an aspect, a system includes one or more controllers configured to receive an image dataset of an actual environment within which the robotic tool is positioned. The controller(s) are also configured to generate a virtual environment of the actual environment based on the image dataset. Further, the controller(s) can control display of the virtual environment including a virtual tool controllable by a user for use to control the robotic tool within the actual environment. The controller(s) can receive user input for altering a perspective view of display of the virtual environment from a first perspective view to a second perspective view. Further, the controller(s) can maintain orientation of display of the virtual tool with respect to the user during display of the first perspective view and the second perspective view of the virtual environment.

What is claimed: 
     
       1. A system comprising:
 a robotic tool interface configured to control a robotic tool; 
 at least one controller configured to:
 receive an image dataset from an imaging system with an imaging perspective of an actual environment within which the robotic tool is positioned; 
 generate a virtual environment of the actual environment based on the image dataset; 
 control display of the virtual environment including a virtual tool controllable by a user for use to control the robotic tool within the actual environment; 
 receive user input for controlling the virtual tool to control the robotic tool; 
 output a command, based on the received input, to control movement of the robotic tool via the robotic tool interface; 
 receive user input for altering a virtual perspective view of display of the virtual environment to another virtual perspective view that does not coincide with the imaging perspective; 
 alter the display of the virtual environment from one of the one or more virtual perspective views to another virtual perspective view; 
 provide simultaneous display of the actual environment from the imaging perspective and of the virtual environment from the other virtual perspective view; and 
 maintain orientation of display of the virtual tool with respect to the user during display of the other virtual perspective view of the virtual environment. 
 
 
     
     
       2. The system of  claim 1 , wherein the image dataset is a three-dimensional (3D) image dataset. 
     
     
       3. The system of  claim 1 , wherein the environment is a surgical environment. 
     
     
       4. The system of  claim 3 , wherein the environment is an ocular surgical environment. 
     
     
       5. The system of  claim 1 , wherein the robotic tool comprises one of forceps, lighting equipment, syringe, scissors, irrigation equipment, suction equipment, scraper, probe, pick, and needle. 
     
     
       6. The system of  claim 1 , wherein the at least one controller is configured to:
 receive the image dataset in real time over a period of time; and 
 generate and change the virtual environment in real time based on the real time receipt of the image dataset. 
 
     
     
       7. The system of  claim 1 , wherein the image dataset comprises one of an optical coherence tomography (OCT) image dataset, a magnetic resonance image (MRI) dataset, computed tomography (CT) image dataset, or an ultrasound image dataset. 
     
     
       8. The system of  claim 1 , wherein the at least one controller is configured to:
 specify one or more points for constraining movement of the robotic tool about the one or more points; and 
 prevent movement of the robotic tool except about the one or more constraint points. 
 
     
     
       9. The system of  claim 8 , wherein the one or more points define a boundary for movement of the robotic tool. 
     
     
       10. The system of  claim 8 , wherein the one or more points define an axis pivot or plane of pivot for constraining translational motion of at least one portion of the robotic tool. 
     
     
       11. The system of  claim 1 , wherein the at least one controller is configured to specify one of a point, axis, or plane for constraining pivot movement of the robotic tool about the one of the point, axis, or plane. 
     
     
       12. The system of  claim 1 , wherein the at least one controller is configured to control a display to alternate between display of the first perspective view and the second perspective view. 
     
     
       13. The system of  claim 1 , wherein the second perspective view of the virtual environment substantially opposes the first perspective view of the virtual environment,
 wherein the at least one controller configured to:
 receive user input for moving the virtual tool in a direction; and 
 move the virtual tool, in both the displayed first perspective and the second perspective, in the direction specified by the received user input for moving the virtual tool. 
 
 
     
     
       14. The system of  claim 1 , wherein the at least one controller comprises at least one processor and memory. 
     
     
       15. A method comprising:
 receiving an image dataset of an actual environment within which a robotic tool is positioned; 
 generating a virtual environment of the actual environment based on the image dataset; 
 displaying the virtual environment including a virtual tool controllable by a user for use to control the robotic tool within the actual environment; 
 receiving user input for controlling the virtual tool to control the robotic tool; 
 controlling movement of the robotic tool based on the received input; 
 outputting a command, based on the received input, to control movement of the robotic tool via the robotic tool interface; 
 receiving user input for altering a perspective view of display of the virtual environment to another virtual perspective view that does not coincide with the imaging perspective; 
 altering the display of the virtual environment from one of the one or more virtual perspective views to another virtual perspective view; 
 providing simultaneous display of the actual environment from the imaging perspective and of the virtual environment from the other virtual perspective view; and 
 maintaining orientation of display of the virtual tool with respect to the user during display of the other virtual perspective view of the virtual environment. 
 
     
     
       16. The method of  claim 15 , wherein the image dataset is a three-dimensional (3D) image dataset. 
     
     
       17. The method of  claim 15 , wherein the environment is a surgical environment. 
     
     
       18. The method of  claim 15 , wherein the environment is an ocular surgical environment. 
     
     
       19. The method of  claim 15 , wherein the robotic tool comprises one of forceps, lighting equipment, syringe, scissors, irrigation equipment, suction equipment, scraper, probe, pick, and needle. 
     
     
       20. The method of  claim 15 , wherein receiving the image dataset comprises receiving the image dataset in real time over a period of time; and
 wherein the method comprises generating and changing the virtual environment in real time based on the real time receipt of the image dataset. 
 
     
     
       21. The method of  claim 15 , wherein the image dataset comprises one of an optical coherence tomography (OCT) image dataset, a magnetic resonance image (MRI) dataset, computed tomography (CT) image dataset, or an ultrasound image dataset. 
     
     
       22. The method of  claim 15 , further comprising:
 specifying one or more points for constraining movement of the robotic tool about the one or more points; and 
 preventing movement of the robotic tool except about the one or more constraint points. 
 
     
     
       23. The method of  claim 15 , wherein the one or more points define a boundary for movement of the robotic tool. 
     
     
       24. The method of  claim 15 , wherein the one or more points define an axis pivot or plane of pivot for constraining translational motion of at least one portion of the robotic tool. 
     
     
       25. The method of  claim 1 , further comprising specifying one of a point, axis, or plane for constraining pivot movement of the robotic tool about the one of the point, axis, or plane. 
     
     
       26. The method of  claim 15 , further comprising controlling a display to alternate between display of the first perspective view and the second perspective view. 
     
     
       27. The method of  claim 15 , wherein the second perspective view of the virtual environment substantially opposes the first perspective view of the virtual environment,
 wherein the method further comprises:
 receiving user input for moving the virtual tool in a direction; and 
 moving the virtual tool, in both the displayed first perspective and the second perspective, in the direction specified by the received user input for moving the virtual tool.