Surgical Cockpit Comprising Multisensory and Multimodal Interfaces for Robotic Surgery and Methods Related Thereto

Abstract

Local surgical cockpits comprising local surgical consoles that can communicate with any desired remote surgical module (surgical robot), for example via a shared Transmission Control Protocol/Internet Protocol (TCP/IP) or other unified open source communication protocol or other suitable communication system. The systems and methods, etc., herein can also comprise a modular approach wherein multiple surgical consoles can network supporting collaborative surgery regardless of the physical location of the surgeons relative to each other and/or relative to the surgical site. Thus, for example, an operator operating a local surgical cockpit can teleoperate using a remote surgical module on a patient in the same room as the surgeon, or surgeons located in multiple safe locations can telemanipulate remote multiple surgical robots on a patient in or close to a war zone.

1 . A local surgical cockpit comprising:
 a base, a frame disposed on the base, a seat for an operator disposed on the frame, and a remote surgical console configured such that the operator can operate the console for remote surgery while in the seat, wherein the seat is ergonomic and is operably connected to the frame such that the seat can be retainably tilted from a substantially upright position to a substantially supine position.   
     
     
         2 . The local surgical cockpit of  claim 1  wherein the seat comprises an independently movable headrest, backrest, seating plate and footrest, the seat further comprising positioning elements operably connected to the independently movable headrest, backrest, seating plate and footrest and providing at least three axes of retainable positioning movement for each of the independently movable headrest, backrest, seating plate and footrest. 
     
     
         3 . The local surgical cockpit of  claim 2  wherein the seat includes a lumbar support comprising retainable positioning movement for support of the lower back. 
     
     
         4 . The local surgical cockpit of  claim 1  or  2  wherein a reference body posture of the seat corresponds to a human body posture that is fully relaxed in micro gravity. 
     
     
         5 . The local surgical cockpit of  claim 1  or  2  wherein the cockpit further comprises at least one peripheral device operably connected to move with the seat when the seat is moved so that the location of the peripheral device relative to the operator in the seat is substantially unchanged. 
     
     
         6 . The local surgical cockpit of  claim 4  wherein the peripheral device is at least one of a monitor facing an operator in the seat and operably linked to display a remote surgical site, a heads-up display disposed in front of the local operator's eyes, and an input device disposed at a hand of the operator and operably linked to provide input to a corresponding device located at the remote surgical site. 
     
     
         7 . The local surgical cockpit of  claim 6  wherein the cockpit comprises all of the monitor, the heads-up display and the input device. 
     
     
         8 . A local surgical cockpit comprising:
 a local surgical cockpit comprising a local surgical console configured for transmitting surgical movements of local operator operating the local surgical console to a remote surgery site, and a head-mounted display disposed in front of the local operator's eyes in surgical position in the cockpit to operate the console for surgery, wherein the head-mounted display is configured to depict at least images of a remote surgical site under remote operation by the operator.   
     
     
         9 . The local surgical cockpit of  claim 8  wherein the local surgical cockpit is part of a system and the system further comprises remote image sensors operably connected to the head-mounted display to transmit the image of the remote surgical site. 
     
     
         10 . The local surgical cockpit of  claim 8  wherein the head-mounted display extends to the local operator's eyes from an articulated boom disposed in front of the local operator's eyes. 
     
     
         11 . The local surgical cockpit of  claim 8  wherein the articulated boom is actuated by at least one hand control located on the cockpit. 
     
     
         12 . The local surgical cockpit of  claim 8  wherein the boom is actuated by voice control. 
     
     
         13 . The local surgical cockpit of  claim 8  wherein the head-mounted display is disposed on a head-mounted frame configured to rest on an operator's head and to maintain the images in front of the local operator's eyes when the operator's head moves. 
     
     
         14 . The local surgical cockpit of  claim 8  wherein the head-mounted display comprises two separate streams of video displayed to each eye of the local operator's eyes, each stream comprising corresponding right and left eye views of a remote surgical site to provide a 3-D image of the site. 
     
     
         15 . The local surgical cockpit of  claim 8  wherein the cockpit further comprises at least one monitor. 
     
     
         16 . A local surgical cockpit comprising:
 a local surgical cockpit comprising a local surgical console configured for transmitting surgical movements of an operator operating the local surgical console to a remote surgery site, and comprising at least one image display device configured to depict at least one image of the remote surgical site, the display device further depicting augmented reality for the operator comprising augmented information shown on the display and superimposed over the image of the remote surgical site.   
     
     
         17 . The local surgical cockpit of  claim 16  wherein the local surgical cockpit is part of a system and the system further comprises remote image sensors operably connected to the head-mounted display to transmit the image of the remote surgical site. 
     
     
         18 . The local surgical cockpit of  claim 16  wherein the augmented information comprises at least one of preselected margins to dissect during the surgery and a mask of vital structures in the remote surgical site overlaid over the images of the remote surgical site. 
     
     
         19 . The local surgical cockpit of  claim 16  or  18  wherein the display device further displays further augmented information either to a side of or superimposed over the image of the remote surgical site and the further augmented information comprises at least one of blood pressure, temperature, O 2  level, CO 2  level, intracranial pressure, a preplanned trajectory for a surgical tool, tool type, suction on/off, a bottom task bar, recording capabilities, current time, and elapsed time. 
     
     
         20 . The local surgical cockpit of  claim 16  or  18  wherein the image of the remote surgical site and the augmented information comprise blending graphical images with real-world views of the remote surgical site. 
     
     
         21 . The local surgical cockpit of  claim 16  or  18  wherein the image of the remote surgical site is provided by at least one of an endoscopic camera, a remote surgical site camera, or a camera showing an operating room. 
     
     
         22 . A local surgical cockpit comprising:
 a local surgical cockpit comprising a local surgical console configured for transmitting surgical movements of an operator operating the local surgical console to a remote surgery site, and comprising a local surgical instrument comprising local input surgical fingers configured to provide input to corresponding remote surgical fingers configured to manipulate a remote surgical instrument at a remote operation site, wherein the local fingers are high frequency haptic fingers configured to provide tactile feedback to the operator based on acceleration of the remote surgical instrument manipulated by the remote surgical fingers.   
     
     
         23 . The local surgical cockpit of  claim 22  wherein the local surgical cockpit is part of a system and the system further comprises the remote surgical fingers, and wherein the remote surgical fingers are haptic fingers configured to provide tactile feedback to the operator based on acceleration of the remote surgical instrument manipulated by the remote surgical fingers. 
     
     
         24 . The local surgical cockpit of  claim 23  wherein the local surgical cockpits of the system are configured such that operators in different locales can operate simultaneously on a single surgical site. 
     
     
         25 . The local surgical cockpit of  claim 24  wherein the local surgical cockpits of the system are configured such that operators can relieve each other in a single surgery at a single surgical site. 
     
     
         26 . The local surgical cockpit of  claim 23  wherein the local surgical cockpits of the system are configured to provide a teaching surgical cockpit and a student surgical cockpit providing haptic feedback to a student operator generated by a teaching operator. 
     
     
         27 . The local surgical cockpit of  claim 23  wherein the haptic feedback to the student comprises movements of a remote surgical instrument controlled by the teaching operator. 
     
     
         28 . The local surgical cockpit of  claim 23  wherein the haptic feedback to the student comprises tactile feedback from a surgical site being operated on by the teaching operator. 
     
     
         29 . A local surgical cockpit comprising:
 a local surgical cockpit comprising a local surgical console configured for transmitting surgical movements of an operator operating the local surgical console to a remote surgery site, and comprising at least seven degrees of freedom for a local surgical instrument manipulated by a robotic arm manipulated by the operator, wherein the console is configured such that the seven degrees of freedom are transmissible to a remote surgical instrument located at a remote surgical site and manipulated by the operator operating the console.   
     
     
         30 . The local surgical cockpit of  claim 29  wherein the local surgical cockpit is part of a system and the system further comprises the remote surgical instrument operably connected to the local surgical instrument such that the remote surgical instrument precisely responds in at least seven corresponding degrees of freedom to movements of the local surgical instrument. 
     
     
         31 . The local surgical cockpit of  claim 29  wherein the degrees of freedom comprise at least nine degrees of freedom for the local surgical instrument manipulated by the operator and a corresponding nine degrees of freedom for the remote surgical instrument. 
     
     
         32 . The local surgical cockpit of  claim 29  wherein the degrees of freedom comprise at least twelve degrees of freedom for the local surgical instrument manipulated by the operator and a corresponding twelve degrees of freedom for the remote surgical instrument, wherein the local robotic arm comprises a shoulder joint, an elbow joint, a wrist joint and the three fingers, each comprising at least the following degrees of freedom: shoulder comprises 2 degrees of freedom; elbow comprises 1 degree of freedom; wrist comprises 3 degrees of freedom; the three fingers comprise 2 degrees of freedom each. 
     
     
         33 . The local surgical cockpit of  claim 29  wherein local surgical instrument comprises at least three input fingers configured to provide input to a corresponding at least three remote surgical fingers configured to manipulate a remote surgical instrument at a remote operation site, wherein the at least three input fingers are configured to be manipulated by a single hand of an operator operating the local surgical instrument, and wherein the at least seven degrees of freedom comprise at least two degrees of freedom for two of the three remote surgical fingers and at least three degrees of freedom for a third of the three remote surgical fingers. 
     
     
         34 . The local surgical cockpit of  claim 29  wherein local surgical instrument comprises at least three input fingers configured to provide input to a corresponding at least three remote surgical fingers configured to manipulate a remote surgical instrument at a remote operation site, wherein the at least three input fingers are configured to be manipulated by a single hand of an operator operating the local surgical instrument, and wherein the degrees of freedom comprise at least nine degrees of freedom comprising at least three degrees of freedom for each of the three remote surgical fingers. 
     
     
         35 . The local surgical cockpit of  claim 29  wherein local surgical instrument comprises at least three input fingers configured to provide input to a corresponding at least three remote surgical fingers configured to manipulate a remote surgical instrument at a remote operation site, wherein the at least three input fingers are configured to be manipulated by a single hand of an operator operating the local surgical instrument, wherein the local robotic arm comprises a shoulder joint, an elbow joint, a wrist joint and the three fingers, each comprising at least the following degrees of freedom: shoulder comprises 2 degrees of freedom; elbow comprises 1 degree of freedom; wrist comprises 3 degrees of freedom; the three fingers comprise 2 degrees of freedom each. 
     
     
         36 . The local surgical cockpit of  claim 35  wherein the three fingers comprise 3 degrees of freedom each. 
     
     
         37 . The local surgical cockpit of  claim 29  wherein the degrees of freedom provide for variable desired positioning and orientation of a tip of the remote surgical instrument in space in 6 parameters including Cartesian position (x,y,z), and angular orientation (x y z θ, θ, θ). 
     
     
         38 . The local surgical cockpit of  claim 29  wherein control of the remote surgical instrument further comprises scaling factors configured such that motion input by the operator is attenuated or amplified with respect to the remote surgical instrument. 
     
     
         39 . The local surgical cockpit of  claim 29  wherein control of the remote surgical instrument further comprises indexing configured to allow the operator to disengage the input device from the remote surgical instrument to reposition his/her arms and engage again. 
     
     
         40 . A local surgical cockpit comprising:
 a local surgical cockpit comprising a local surgical console configured for transmitting surgical movements of an operator operating the local surgical console to a remote surgery site, and comprising a local surgical instrument comprising at least three input fingers configured to provide input to a corresponding at least three remote surgical fingers configured to manipulate a remote surgical instrument at a remote operation site, wherein the at least three input fingers are configured to be manipulated by a single hand of an operator operating the local surgical instrument.   
     
     
         41 . The local surgical cockpit of  claim 40  wherein the local surgical cockpit is part of a system and the system further comprises the three remote surgical fingers operably connected to the three input fingers such that the three remote surgical fingers precisely respond to movements of the three input fingers. 
     
     
         42 . The local surgical cockpit of  claim 40  wherein the at least three input fingers are configured to correspond respectively to a) an operator's thumb, b) an operator's index and middle fingers, and c) an operator's ring and little fingers. 
     
     
         43 . The local surgical cockpit of  claim 40  wherein the at least three input fingers are configured to correspond respectively to a) an operator's thumb, b) an operator's index finger, and c) an operator's middle, ring and little fingers. 
     
     
         44 . The local surgical cockpit of  claim 40  wherein the at least three input fingers are haptic fingers configured to provide tactile feedback to the operator based on acceleration of a remote surgical instrument manipulated by the remote surgical fingers. 
     
     
         45 . The local surgical cockpit of  claim 40  wherein the three input fingers are operably connected so that two fingers control remote surgical fingers and the remaining third finger controls an external device. 
     
     
         46 . The local surgical cockpit of  claim 40  wherein the external device is a one or more of an electrocautery device, a laser photocoagulator, a staple applier. 
     
     
         47 . The local surgical cockpit of  claim 40  wherein the external device is an optical aspect of the camera system such as focus, zoom, rotation, or field-of-view. 
     
     
         48 . A local surgical cockpit comprising:
 a local surgical cockpit comprising a local surgical console configured for transmitting surgical movements of an operator operating the local surgical console to a remote surgery site, and comprising at least one haptic foot pedal configured to be operably connected to at least one remote device at a remote operation site, wherein the at least one haptic foot pedal is configured to be manipulated by a foot of the operator operating the local surgical console to cause a movement or control change in the remote device.   
     
     
         49 . The local surgical cockpit of  claim 48  wherein the local surgical cockpit is part of a system and the system further comprises the at least one remote device operably connected to the at least one haptic foot pedal. 
     
     
         50 . The local surgical cockpit of  claim 48  wherein the local surgical cockpit further comprises at least two haptic foot pedals configured to be operably connected to the at least one remote device at the remote operation site. 
     
     
         51 . The local surgical cockpit of  claim 48  wherein the local surgical cockpit further comprises at least eight haptic foot pedals configured to be operably connected to at least two remote devices at the remote operation site, the at least eight haptic foot pedals divided to provide at least a first foot pedal set and second foot pedal set, wherein a first foot pedal set and second foot pedal set are each disposed to be manipulated by a right foot of the operator and by a left foot of the operator, respectively. 
     
     
         52 . The local surgical cockpit of  claim 51  wherein, for each of the first foot pedal set and second foot pedal set, the sets each contain four pedals with each of the four pedals in one of four quadrants of a circle. 
     
     
         53 . The local surgical cockpit of  claim 51  wherein opposed pairs of the four pedals are assigned opposed functions at the remote surgical site. 
     
     
         54 . The local surgical cockpit of  claim 51  wherein the opposed functions are suction and irrigation. 
     
     
         55 . The local surgical cockpit of  claim 51  wherein each of the four pedals is assigned complementary functions for a remote instrument at the remote surgical site. 
     
     
         56 . The local surgical cockpit of  claim 55  wherein each of the four pedals controls the viewing angles of an endoscopic camera. 
     
     
         57 . The local surgical cockpit of  claim 48  wherein the local surgical cockpit further comprises a dead zone that prevents two opposing functions being implemented simultaneously. 
     
     
         58 . The local surgical cockpit of  claim 48  wherein the at least one haptic foot pedal controls at least one of camera angle, camera zoom, camera focus, irrigation, suction, robot brakes, electric coagulation, laser photocoagulation. 
     
     
         59 . A local surgical cockpit comprising:
 a local surgical cockpit comprising a local surgical console configured for transmitting surgical movements of an operator operating the local surgical console to a remote surgery site, and comprising at least one virtual console control knob presented virtually to the operator and configured to be manipulated by the operator to generate control signals for a corresponding remote device at a remote operation site.   
     
     
         60 . The local surgical cockpit of  claim 59  wherein the local surgical cockpit is part of a system and the system further comprises the corresponding remote device. 
     
     
         61 . The local surgical cockpit of  claim 60  wherein the at least one virtual console control knob is a binary switch configured to provide on/off signals to the corresponding remote device. 
     
     
         62 . The local surgical cockpit of  claim 60  wherein the at least one virtual console control knob is a gradual control knob configured to provide gradual control signals to the corresponding remote device. 
     
     
         63 . The local surgical cockpit of  claim 59  wherein the virtual control knob is operably connected to one of the three fingers of the haptic device. 
     
     
         64 . The local surgical cockpit of  claim 59  wherein the virtual control knob must be virtually gripped by two or more fingers of the haptic device before it may be rotated. 
     
     
         65 . A local surgical cockpit system comprising:
 at least two local surgical cockpits each comprising a surgical console configured for transmitting surgical movements of an operator operating the local surgical console to a distant remote surgery site located outside at least one building containing at least one of the surgical cockpits, wherein each cockpit comprises a respective first and second set of at least two local robotic input arms configured to provide input to corresponding first and second remote sets of at least two corresponding remote robotic arms each configured to manipulate a remote surgical instrument at a single remote operation site, wherein the respective first and second set of local robotic input arms are configured to be manipulated by respective first and second operators working in concert on the remote surgical site.   
     
     
         66 . The local surgical cockpit system of  claim 65  wherein the local surgical cockpit system is part of a further system and the further system further comprises the first and second remote sets of at least two corresponding remote robotic arms. 
     
     
         67 . The local surgical cockpit system of  claim 65  wherein the distant remote surgery site is located outside any building containing any of the local surgical cockpits. 
     
     
         68 . The local surgical cockpit of  claim 65  wherein the local surgical cockpits of the system are configured such that operators in different locales can operate simultaneously on a single surgical site. 
     
     
         69 . The local surgical cockpit of  claim 65  wherein the local surgical cockpits of the system are configured such that operators can relieve each other in a single surgery at a single surgical site. 
     
     
         70 . The local surgical cockpit of  claim 65  wherein the local surgical cockpits of the system are configured to provide a teaching surgical cockpit and a student surgical cockpit providing haptic feedback to a student operator generated by a teaching operator. 
     
     
         71 . The local surgical cockpit of  claim 70  wherein the haptic feedback to the student comprises movements of a remote surgical instrument controlled by the teaching operator. 
     
     
         72 . The local surgical cockpit of  claim 70  wherein the haptic feedback to the student comprises tactile feedback from a surgical site being operated on by the teaching operator. 
     
     
         73 . A local surgical cockpit comprising:
 a local surgical cockpit comprising a local surgical console configured for transmitting surgical movements of an operator operating the local surgical console to a remote surgery site, and comprising at least four local robotic input arms configured to provide input to a corresponding at least four remote robotic arms each configured to manipulate a remote surgical instrument at a remote operation site, wherein the at least four local robotic input arms are configured to be manipulated by at least one operator operating the local surgical console.   
     
     
         74 . The local surgical cockpit of  claim 73  wherein the local surgical cockpit is part of a system and the system further comprises the four remote robotic arms operably connected to the four local robotic input arms such that the four remote robotic arms precisely respond to movements of the four local robotic input arms. 
     
     
         75 . The local surgical cockpit of  claim 73  wherein the four remote robotic arms are held in a sole arm-retention structure. 
     
     
         76 . The local surgical cockpit of  claim 75  wherein the sole arm-retention structure is configured to hold the four remote robotic arms such that the arms cannot collide with each other. 
     
     
         77 . The local surgical cockpit of  claim 73  wherein the cockpit is part of a system comprising at least two local surgical cockpits each configured for an operator, and wherein the system is configured such that each operator can simultaneously hold a single remote robotic arm. 
     
     
         78 . The local surgical cockpit of  claim 73  wherein the cockpit is part of a system comprising at least two local surgical cockpits each configured for an operator, and wherein the system is configured such that the operators can switch control of a remote robotic arm between each other. 
     
     
         79 . A local surgical cockpit comprising:
 a local surgical cockpit comprising a local surgical console configured for transmitting surgical movements of an operator operating the local surgical console to a remote surgery site, and comprising local 3-dimensional audio configured to obtain remote 3-dimensional audio input from a remote operation site and provide corresponding local 3-dimensional audio to an operator operating the console.   
     
     
         80 . The local surgical cockpit of  claim 79  wherein the local surgical cockpit is part of a system and the system further comprises remote 3-dimensional audio sensors operably connected to the local 3-dimensional audio such that the local 3-dimensional audio precisely transmit 3-dimensional audio signals from the remote 3-dimensional audio sensors. 
     
     
         81 . The local surgical cockpit of  claim 79  wherein the 3-dimensional audio signals are correlated with tactile feedback to provide correlated response to haptic input devices at the local surgical cockpit.