Optical axis calibration of robotic camera system

Abstract

A method, instructions for which are executed from a computer-readable medium, calibrates a robotic camera system having a digital camera connected to an end-effector of a serial robot. The end-effector and camera move within a robot motion coordinate frame (“robot frame”). The method includes acquiring, using the camera, a reference image of a target object on an image plane having an optical coordinate frame, and receiving input signals, including a depth measurement and joint position signals. Separate roll and pitch offsets are determined of a target point within the reference image with respect to the robot frame while moving the robot. Offsets are also determined with respect to x, y, and z axes of the robot frame while moving the robot through another motion sequence. The offsets are stored in a transformation matrix, which is used to control the robot during subsequent operation of the camera system.

What is claimed: 
     
       1. A method for calibrating a robotic camera system having a digital camera that is integral with an ophthalmic microscope and connected to an end-effector of a serial robot, wherein the end-effector and the digital camera move within a robot motion coordinate frame, the method comprising:
 acquiring, using the digital camera, a reference image of a target object in an image plane having an optical coordinate frame, the target object including an eye of a human patient; 
 receiving input signals via an electronic control unit (ECU) in communication with the serial robot and the digital camera, the input signals including a depth measurement indicative of a linear distance between the digital camera and the target object, and a set of joint position signals collectively describing a position of the digital camera within the robot motion coordinate frame, the robot motion coordinate frame having an x-axis, a y-axis, and a z-axis; 
 determining, via the ECU, a roll offset and a pitch offset of a target point within the reference image with respect to the robot motion coordinate frame while moving the serial robot through a first calibrated motion sequence; 
 determining, via the ECU after determining the roll offset and the pitch offset, each of an x-axis offset, a y-axis offset, and a z-axis offset of the target point with respect to the robot motion coordinate frame while moving the serial robot through a second calibrated motion sequence; 
 storing the roll offset, the pitch offset, the x-axis offset, the y-axis offset, and the z-axis offset in a transformation matrix within memory of the ECU; and 
 controlling a third motion sequence of the serial robot, via the ECU, during a subsequent operation of the robotic camera system using the transformation matrix, the subsequent operation of the robotic camera system being performed using the ophthalmic microscope as part of an eye surgery of the eye of the human patient. 
 
     
     
       2. The method of  claim 1 , wherein the digital camera has a variable optical working distance between the digital camera and the target object controlled by a focus motor, wherein determining the z-axis offset includes extracting the z-axis offset from a lookup table indexed by the variable optical working distance and a rotary position or encoder count of the focus motor. 
     
     
       3. The method of  claim 2 , further comprising:
 populating the lookup table via the ECU while controlling the focus motor through a focal range corresponding to the variable optical working distance. 
 
     
     
       4. The method of  claim 1 , further comprising:
 receiving an autofocus setting of the digital camera via the ECU; and 
 processing the autofocus setting of the robotic camera system via the ECU to determine the depth measurement. 
 
     
     
       5. The method of  claim 1 , further comprising:
 measuring the depth measurement using a depth sensor; and 
 measuring the joint position signals via a corresponding set of joint position sensors of the serial robot. 
 
     
     
       6. The method of  claim 5 , wherein measuring the depth measurement using a depth sensor is performed using a laser distance meter or an optical sensor. 
     
     
       7. The method of  claim 1 , wherein acquiring the reference image of the target object includes collecting a digital image of a two-dimensional checkerboard graphic using the digital camera. 
     
     
       8. The method of  claim 1 , further comprising:
 displaying three-dimensional images of the target object via one or more display screens during the subsequent operation. 
 
     
     
       9. A robotic camera system comprising:
 an ophthalmic microscope that is integral with a digital camera and connectable to an end-effector of a serial robot, wherein the end-effector and digital camera move within a robot motion coordinate frame; and 
 an electronic control unit (ECU) in communication with the digital camera, and configured to:
 acquire, using the digital camera, a reference image of a target object in an image plane having an optical coordinate frame, the target object including an eye of a human patient; 
 receive input signals, including a depth measurement indicative of a linear distance between the digital camera and the target object, and a set of joint position signals collectively describing a position of the digital camera in the robot motion coordinate frame, the robot motion coordinate frame having an x-axis, a y-axis, and a z-axis; 
 determine a roll offset and a pitch offset of a target point within the reference image with respect to the robot motion coordinate frame while moving the serial robot through a first calibrated motion sequence; 
 determine, after determining the roll offset and the pitch offset, each of an x-axis offset, a y-axis offset, and a z-axis offset of the target point with respect to the robot motion coordinate frame while moving the serial robot through a second calibrated motion sequence; 
 store the roll offset, the pitch offset, the x-axis offset, the y-axis offset, and the z-axis offset in a transformation matrix within memory of the ECU; and 
 control a third motion sequence of the serial robot using the transformation matrix during a subsequent operation of the robotic camera system, wherein the subsequent operation of the robotic camera system is performed using the ophthalmic microscope as part of an eye surgery of the eye of the human patient. 
 
 
     
     
       10. The robotic camera system of  claim 9 , further comprising the serial robot. 
     
     
       11. The robotic camera system of  claim 9 , wherein the digital camera includes a focus motor, and has a variable optical working distance between the digital camera and the image plane that is controlled by the focus motor, wherein the ECU is configured to extract the z-axis offset from a lookup table indexed by the variable optical working distance and a rotary position or encoder count of the focus motor. 
     
     
       12. The robotic camera system of  claim 11 , wherein the ECU is configured to populate the lookup table while controlling the focus motor through a focal range corresponding to the variable optical working distance. 
     
     
       13. The robotic camera system of  claim 9 , wherein the ECU is configured to receive an autofocus setting of the digital camera, and to determine the depth measurement using the autofocus setting. 
     
     
       14. The robotic camera system of  claim 9 , further comprising a depth sensor operable for determining the depth measurement. 
     
     
       15. The robotic camera system of  claim 9 , wherein the ECU is configured to acquire the reference image of the target object by collecting a digital image of a two-dimensional checkerboard graphic using the digital camera. 
     
     
       16. The robotic camera system of  claim 14 , wherein the depth sensor includes a laser distance meter or an optical sensor. 
     
     
       17. The robotic camera system of  claim 9 , further comprising one or more display screens, wherein the ECU is configured to display three-dimensional images of the target object via the one or more display screens during the subsequent operation. 
     
     
       18. A non-transitory computer-readable medium on which is recorded instructions, execution of which by a processor causes the processor, when used with a robot camera system having a digital camera connected to an end-effector of a serial robot, to:
 acquire, from a digital camera, that is integral with an ophthalmic microscope, a reference image of a target object on an image plane having an optical coordinate frame, the target object including an eye of a human patient; 
 receive input signals, including a depth measurement indicative of a linear distance between the digital camera and the target object, and a set of joint position signals collectively describing a position of the digital camera in a robot motion coordinate frame, the robot motion coordinate frame having an x-axis, a y-axis, and a z-axis; 
 determine a roll offset and a pitch offset of a target point within the reference image with respect to the robot motion coordinate frame while moving the serial robot through a first calibrated motion sequence; 
 determine, after determining the roll offset and the pitch offset, each of an x-axis offset, a y-axis offset, and a z-axis offset of the target point with respect to the robot motion coordinate frame while moving the serial robot through a second calibrated motion sequence; 
 store the roll offset, the pitch offset, the x-axis offset, the y-axis offset, and the z-axis offset in a transformation matrix within the computer-readable medium, thereby calibrating a robotic camera system having the digital camera; and 
 control a third motion sequence of the serial robot during a subsequent operation of the robotic camera system use the transformation matrix to control, wherein the subsequent operation of the robotic camera system is performed using the ophthalmic microscope as part of an eye surgery of the eye of the human patient. 
 
     
     
       19. The non-transitory computer-readable medium of  claim 18 , wherein the digital camera is a stereoscopic camera, and the execution of the instructions by the processor causes the processor to:
 display three-dimensional images of the target object via one or more display screens during the subsequent operation. 
 
     
     
       20. The non-transitory computer-readable medium of  claim 18 , wherein the digital camera has a variable optical working distance between the digital camera and the image plane that is controlled by a focus motor, and wherein execution of the instructions by the processor causes the processor to:
 determine a rotary position or encoder count the focus motor; and 
 extract the z-axis offset from a lookup table populated by the z-axis offset and indexed by the variable optical working distance and the rotary position or encoder count of the focus motor.