Teleoperation for training of robots using machine learning

Abstract

Methods and systems for using a teleoperation system to train a robot to perform tasks using machine learning are described herein. A teleoperation system may be used to record actions of a robot as used by a human teleoperator. The teleoperation system may provide a teleoperator insight into the state of the robot and may provide feedback to the teleoperator allowing the teleoperator to feel what the robot is feeling. For example, sensor information from the robot may be sent to the teleoperation system and output to the teleoperator in various forms including vibrations, video, visual cues, or sound. The teleoperation system may output visual guides to the teleoperator so that the teleoperator may know how to control the robot to complete a task in a desired manner.

What is claimed is: 
     
         1 . A system comprising a tangible, non-transitory, machine-readable medium storing instructions that when executed by one or more processors effectuate operations comprising:
 obtaining, with a computer system, a plurality of records of one or more humans teleoperating one or more robots, the plurality of records comprising:
 outputs from sensors of the one or more robots indicative of states and environments of the one or more robots, and 
 commands to the one or more robots, wherein the commands are generated based on teleoperation inputs obtained from humans upon being presented with the outputs; and 
   training, with the computer system, a reinforcement-learning model on the plurality of records to mimic the commands to the one or more robots given new inputs from the sensors of the one or more robots.   
     
     
         2 . The system of  claim 1 , wherein the operations further comprise:
 re-training, with the computer system, the trained reinforcement-learning model to control a robot without teleoperation from humans; and   storing, with the computer system, the re-trained reinforcement-learning model in memory, wherein training the reinforcement learning model comprises:
 determining associations between actions and rewards indicated in the plurality of records; and 
 adjusting one or more weights of the reinforcement learning model based on the determined associations. 
   
     
     
         3 . The system of  claim 2 , wherein the re-training comprises:
 determining, based on a reinforcement learning policy, an action that is different from an action indicated by the plurality of records;   causing a first robot of the one or more robots to perform the action; and   in response to causing the first robot to perform the action, adjusting one or more weights of the reinforcement learning model.   
     
     
         4 . The system of  claim 1 , wherein obtaining a plurality of records comprises:
 causing movement, based on input received from a teleoperator, an arm of a first robot of the one or more robots;   detecting, via the arm of the first robot, contact with an object; and   in response to detecting contact with the object, outputting haptic feedback to the teleoperator.   
     
     
         5 . The system of  claim 4 , wherein outputting haptic feedback comprises outputting vibrations via a glove that is worn by the teleoperator. 
     
     
         6 . The system of  claim 1 , wherein obtaining a plurality of records comprises:
 causing, based on input received from a teleoperator, movement of an arm of a first robot of the one or more robots, wherein the movement is in a first direction;   determining, via information from a sensor of the arm of the first robot, that the arm should not be moved further in the first direction; and   in response to determining that the arm should not be moved further in the first direction, outputting feedback to the teleoperator.   
     
     
         7 . The system of  claim 6 , wherein outputting feedback to the teleoperator comprises outputting a notification to a display of an augmented reality headset worn by the teleoperator. 
     
     
         8 . The system of  claim 6 , wherein outputting feedback to the teleoperator comprises outputting vibrations to a device worn on a shoulder of the teleoperator. 
     
     
         9 . The system of  claim 6 , wherein outputting feedback to the teleoperator comprises outputting vibrations to a control device operated by the teleoperator. 
     
     
         10 . The system of  claim 1 , wherein obtaining the plurality of records comprises:
 receiving first input indicating movement for the robot to perform;   receiving second input indicating that the first input does not satisfy one or more criteria; and   in response to receiving the second input, associating the first input with a negative reward in the reinforcement-learning model.   
     
     
         11 . The system of  claim 1 , wherein obtaining a plurality of records further comprises:
 obtaining, from a plurality of cameras of a first robot of the one or more robots, video of an environment associated with the first robot and depth information associated with the video; and   outputting the video on a display of a headset, wherein a first portion of the video is output to a left eye view of the headset and a second portion of the video is output to a right eye view of the headset, and wherein the depth information is overlayed on the video.   
     
     
         12 . The system of  claim 1 , wherein obtaining a plurality of records further comprises:
 obtaining an indication of a task for a first robot of the one or more robots to perform; obtaining video from a plurality of cameras of the first robot;   obtaining second video information associated with a successful completion of the task;   generating a visual guide indicating a plurality of actions to perform to complete the task, and locations where each action of the plurality of actions should be performed; and   outputting, on a display associated with a teleoperator of the first robot, the visual guide onto the first video, wherein a first portion of the visual guide is shown in a corresponding location in the first video.   
     
     
         13 . The system of  claim 11 , wherein the first portion of the video is recorded via a left-side camera of the robot and the second portion of the video is recorded via a right-side camera of the robot. 
     
     
         14 . The system of  claim 1 , wherein obtaining the plurality of records comprises:
 outputting data corresponding to a first robot of the one or more robots on a headset display;   receiving input from a teleoperator of the first robot indicating a movement for the robot to perform; and   based on receiving the input from the teleoperator, outputting updated data on the headset display.   
     
     
         15 . The system of  claim 14 , wherein the system comprises a robot that the trained reinforcement-learning model is configured to control. 
     
     
         16 . The system of  claim 1 , wherein obtaining the plurality of records comprises:
 obtaining task information indicating an object for a first robot of the one or more robots to manipulate;   obtaining video from a plurality of cameras of the first robot, wherein the video comprises a view of the object;   determining, based on inputting the video into a machine learning model that has been trained on previous recordings of teleoperators performing a task, that the object is not oriented correctly; and   in response to determining that the object is not oriented correctly, outputting an image of the object in a desired orientation, wherein the image is overlayed onto the video at a location indicating where the object should be moved to by the first robot.   
     
     
         17 . The system of  claim 1 , wherein obtaining a plurality of records further comprises:
 obtaining video from a plurality of cameras of a first robot of the one or more robots;   obtaining sensor information from a plurality of sensors of the first robot, wherein the sensor information comprises an indication of a position of a joint of the first robot; and   outputting the video overlayed with the sensor information to a display associated with a teleoperator of the first robot.   
     
     
         18 . The system of  claim 17 , wherein the sensor information further comprises:
 an indication of motor temperature of a motor of the first robot; and   a number of hours that the first robot has been in use since the first robot was last turned off.   
     
     
         19 . The system of  claim 1 , wherein obtaining a plurality of records further comprises:
 obtaining video from a plurality of cameras of a first robot of the one or more robots;   obtaining first sensor information indicating that one or more parts of the first robot is functioning as expected;   in response to obtaining first sensor information indicating that one or more parts of the first robot is functioning as expected, outputting the video overlayed with a user interface element indicating that the one or more parts of the first robot are functioning as expected;   obtaining second sensor information indicating that a portion of the first robot is not functioning as expected; and   in response to obtaining the second sensor information, outputting the video overlayed with an indication of the portion of the first robot that is not functioning as expected.   
     
     
         20 . The system of  claim 1 , further comprising the one or more robots, wherein each of the one or more robots each have more than six degrees of freedom. 
     
     
         21 . The system of  claim 1 , further comprising the one or more robots, wherein a first robot of the one or more robots comprises two arms, each arm of the two arms having a hand, and wherein the first robot is tendon driven, and wherein the first robot has more than 30 degrees of freedom. 
     
     
         22 . A method comprising:
 obtaining, with a computer system, a plurality of records of one or more humans teleoperating one or more robots, the plurality of records comprising:
 outputs from sensors of the one or more robots indicative of states and environments of the one or more robots, and 
 commands to the one or more robots, wherein the commands are generated based on teleoperation inputs obtained from humans upon being presented with the outputs; and 
   training, with the computer system, a reinforcement-learning model on the plurality of records to mimic the commands to the one or more robots given new inputs from the sensors of the one or more robots.