Robot System for Controlling a Robot in a Tele-Operation

Abstract

A robot system controls a robot in a tele-operation. The system includes a robot and a remote gaze-controlled camera (GCC) system. The robot includes one or more manipulators and a vision system that provides a remote operator a close up view of a work area of the robot from the robot's perspective. The robot and the one or more manipulators are remotely controlled by the hands of the operator. The GCC system remotely controls the vision system of the robot. The GCC system relieves the operator of additional manual control of the video camera and the cognitive workload of manual control of the video camera. The GCC system includes a video display for remotely displaying video from the vision system to the operator, an eyetracker for determining an eye/head activity variable of the operator's eye, and a processor that remotely controls the video camera based on the eye/head activity variable.

What is claimed is: 
     
         1 . A robot system for controlling a robot in a tele-operation, comprising:
 a robot that includes one or more manipulators and a vision system that provides a remote operator a close up view of a work area of the robot from the robot's perspective, wherein a body of the robot and the one or more manipulators are remotely controlled by the hands of the operator, wherein the vision system includes a video camera and an actuator that is connected to the video camera and controls at least one setting of the video camera; and   a gaze-controlled camera system for remotely controlling the vision system of the robot, wherein the gaze-controlled camera system relieves the operator of additional manual control of the video camera and the cognitive workload of manual control of the video camera, wherein the gaze-controlled camera system includes a video display that displays remote video from the video camera of the vision system of the robot to an operator of the robot, an eyetracker that images at least one eye of the operator as the operator is observing the video display, measures a change in an image of the at least one eye of the operator over time, and calculates an eye/head activity variable from the measured change in the image, and a processor that is in remote communication with the actuator and the video camera of the vision system of the robot, that translates the eye/head activity variable into a camera control setting of the video camera of the vision system of the robot, and that remotely instructs the actuator of the vision system of the robot to apply the camera control setting to the video camera of the vision system of the robot.   
     
     
         2 . The robot system of  claim 1 , wherein the vision system of the robot further includes a second video camera also controlled by the actuator so that the video camera and the second video camera provide a stereoscopic display of the three-dimensional (3-D) work area of the robot and the eyetracker includes a binocular eyetracker that images both eyes of the operator, measures changes in the images of both eyes, and calculates an eye/head activity variable from the measured change in the images of both eyes, and wherein the processor translates the eye/head activity variable into a camera control setting of the video camera and the second video camera of the vision system of the robot and remotely instructs the actuator of the vision system of the robot to apply the camera control setting to the video camera and the second video camera of the vision system of the robot. 
     
     
         3 . The robot system of  claim 2 , wherein the eye/head activity variable comprises positions of both eyes of the operator with respect to the stereoscopic display and the eye positions are used to calculate the pan, tilt and/or roll angle of the operator's head. 
     
     
         4 . The robot system of  claim 2 , wherein the eye/head activity variable comprises an eye gaze convergence/parallax of the both eyes of the operator. 
     
     
         5 . The robot system of  claim 2 , wherein the camera control setting comprises a camera body separation between the video camera and the second video camera. 
     
     
         6 . The robot system of  claim 2 , wherein the camera control setting comprises a tow-in angle or camera parallax between the video camera and the second video camera. 
     
     
         7 . The robot system of  claim 1 , wherein the eye/head activity variable comprises an eye gaze direction. 
     
     
         8 . The robot system of  claim 1 , wherein the eye/head activity variable comprises an eye pupil diameter. 
     
     
         9 . The robot system of  claim 1 , wherein the eye/head activity variable comprises the at least one eye's horizontal, vertical or longitudinal position with respect to the video display and the at least one eye's positions are used to represent the horizontal, vertical or longitudinal position of the operator's head. 
     
     
         10 . The robot system of  claim 1 , wherein the camera control setting comprises a pan, tilt, or roll angle of the video camera. 
     
     
         11 . The robot system of  claim 1 , wherein the camera control setting comprises a horizontal, vertical, or longitudinal position of the video camera. 
     
     
         12 . The robot system of  claim 1 , wherein the camera control setting comprises a zoom of the video camera. 
     
     
         13 . The robot system of  claim 1 , wherein the camera control setting comprises a focus the video camera. 
     
     
         14 . The robot system of  claim 1 , wherein the camera control setting comprises an illumination level for a scene being viewed by the video camera. 
     
     
         15 . The robot system of  claim 1 , wherein the camera control setting comprises an iris diameter of the video camera. 
     
     
         16 . A robot system for controlling a robot through the head or eye motion of a remote operator, comprising:
 a robot that includes a vision system, wherein the vision system includes a video camera and an actuator that is connected to the video camera and controls at least one setting of the video camera; and   a gaze-controlled camera system for remotely controlling the vision system of the robot, wherein the gaze-controlled camera system includes a video display that displays remote video from the video camera of the vision system of the robot to an operator of the robot, an eyetracker that images at least one eye of the operator as the operator is observing the video display, measures a change in an image of the at least one eye of the operator over time, and calculates an eye/head activity variable from the measured change in the image, and a processor that is in remote communication with the actuator and the video camera of the vision system of the robot, that translates the eye/head activity variable into a camera control setting of the video camera of the vision system of the robot, and that remotely instructs the actuator of the vision system of the robot to apply the camera control setting to the video camera of the vision system of the robot.   
     
     
         17 . The robot system of  claim 16 , wherein the robot further includes one or more manipulators and wherein the body of the robot and the one or more manipulators are remotely controlled by the hands of the operator so that the gaze-controlled camera system relieves the operator of additional manual control of the video camera and the cognitive workload of manual control of the video camera. 
     
     
         18 . The robot system of  claim 17 , wherein the robot, the one or more manipulators and the gaze-controlled camera system are used in a tele-operation. 
     
     
         19 . The robot system of  claim 16 , wherein the vision system of the robot further includes a second video camera also controlled by the actuator so that the video camera and the second video camera provide a stereoscopic display of a real three-dimensional (3_D) space and the eyetracker comprises a binocular eyetracker that images both eyes of the operator, measures changes in the images of both eyes, and calculates an eye/head activity variable from the measured change in the images of both eyes, and wherein the processor translates the eye/head activity variable into a camera control setting of the video camera and the second video camera of the vision system of the robot and remotely instructs the actuator of the vision system of the robot to apply the camera control setting to the video camera and the second video camera of the vision system of the robot. 
     
     
         20 . The robot system of  claim 19 , wherein the eye/head activity variable comprises positions of both eyes of the operator with respect to the stereoscopic display and the eye positions are used to calculate the pan, tilt and/or roll angle of the operator's head. 
     
     
         21 . The robot system of  claim 19 , wherein the eye/head activity variable comprises an eye gaze convergence/parallax of the both eyes of the operator. 
     
     
         22 . The robot system of  claim 19 , wherein the camera control setting comprises a tow-in angle or camera parallax between the video camera and the second video camera. 
     
     
         23 . The robot system of  claim 19 , wherein the camera control setting comprises a camera body separation between the video camera and the second video camera. 
     
     
         24 . The robot system of  claim 16 , wherein the eye/head activity variable comprises an eye gaze direction. 
     
     
         25 . The robot system of  claim 16 , wherein the eye/head activity variable comprises an eye pupil diameter. 
     
     
         26 . The robot system of  claim 16 , wherein the eye/head activity variable comprises the at least one eye's horizontal, vertical or longitudinal position with respect to the video display and the at least one eye's positions are used to represent the horizontal, vertical or longitudinal position of the operator's head. 
     
     
         27 . The robot system of  claim 16 , wherein the camera control setting comprises a pan, tilt, or roll angle of the video camera. 
     
     
         28 . The robot system of  claim 16 , wherein the camera control setting comprises a horizontal, vertical, or longitudinal position of the video camera. 
     
     
         29 . The robot system of  claim 16 , wherein the camera control setting comprises a zoom of the video camera. 
     
     
         30 . The robot system of  claim 16 , wherein the camera control setting comprises a focus the video camera. 
     
     
         31 . The robot system of  claim 16 , wherein the camera control setting comprises an illumination level for a scene being viewed by the video camera. 
     
     
         32 . The robot system of  claim 16 , wherein the camera control setting comprises an iris diameter of the video camera. 
     
     
         33 . A method for controlling a robot through the head or eye motion of a remote operator comprising:
 displaying remote video from a video camera of a vision system of a robot to an operator of the robot using a video display of a gaze-controlled camera system for remotely controlling the vision system of the robot, wherein the vision system includes the video camera and an actuator that is connected to the video camera and controls at least one setting of the video camera and wherein the gaze-controlled camera system includes the video display, an eyetracker, and a processor;   imaging at least one eye of the operator as the operator is observing the video display and measuring a change in an image of the at least one eye of the operator over time using the eyetracker;   calculating an eye/head activity variable from the measured change in the image using the eyetracker,   translating the eye/head activity variable into a camera control setting of the video camera of the vision system of the robot using the processor; and   remotely instructing the actuator of the vision system of the robot to apply the camera control setting to the video camera of the vision system of the robot using the processor.