Teleoperating Of Robots With Tasks By Mapping To Human Operator Pose

Abstract

A system enables teleoperation of a robot based on a pose of a subject. The system includes an image capturing device and an operator system controller that are remotely located from a robotic system controller and a robot. The image capturing device captures images of the subject. The operator system controller maps a processed version of the captured image to a three-dimensional skeleton model of the subject and generates body pose information of the subject in the captured image. The robotic system controller communicates with the operator system controller over a network. The robotic system controller generates a plurality of kinematic parameters for the robot and causes the robot to take a pose corresponding to the pose of the subject in the captured image.

1 . A method for training comprising:
 receiving, by one or more processors, teleoperator data corresponding to instructions for a robot to complete a first task;   receiving, by the one or more processors, sensor data corresponding to an environment surrounding the robot; and   training, by the one or more processors using the received teleoperator data and sensor data, a machine learning algorithm to predict future instructions for one or more robots to complete the first task.   
     
     
         2 . The method of  claim 1 , further comprising:
 capturing, by the robot, the sensor data; and   transmitting, by the robot, the sensor data to the teleoperator.   
     
     
         3 . The method of  claim 2 , further comprising generating the teleoperator data using an operator system. 
     
     
         4 . The method of  claim 3 , further comprising performing the instructions, by the robot, after receiving the generated teleoperator data from the operator system. 
     
     
         5 . The method of  claim 1 , wherein the machine learning algorithm is a deep learning model and/or a neural network. 
     
     
         6 . The method of  claim 1 , further comprising receiving an identification of the first task, wherein training the machine learning algorithm further includes training the machine learning algorithm to predict the future instructions using the received identification of the first task. 
     
     
         7 . The method of  claim 1 , wherein the sensor data includes an image of the environment surrounding the robot. 
     
     
         8 . The method of  claim 1 , wherein the sensor data comprises an image of an object to be manipulated by the robot to complete the first task, and wherein training the machine learning algorithm further includes training the machine learning algorithm to predict the future instructions using the image of the object. 
     
     
         9 . The method of  claim 1 , further comprising:
 receiving motion trajectory information corresponding to a sequence of movements performed by the robot to complete the first task in response to the instructions, wherein training the machine learning algorithm to predict the future instructions includes training the machine learning algorithm to predict a series of movements for the one or more robots to complete the first task.   
     
     
         10 . The method of  claim 1 , wherein the sensor data includes an image of one or more objects to be manipulated by the robot and an environment surrounding the robot prior to completion of the first task. 
     
     
         11 . The method of  claim 1 , wherein training the machine learning algorithm comprises:
 calculating a difference between the predicted future instructions and the received instructions; and   adjusting, based on the difference, one or more coefficients of the machine learning algorithm to reduce the difference.   
     
     
         12 . A system for training a machine learning algorithm comprising:
 one or more computing devices storing an imitation learning engine, the imitation learning engine configured to:
 receive teleoperator data corresponding to instructions performed by a robot to complete a first task; 
 receive sensor data corresponding to an environment surrounding the robot; and 
 train, using the received teleoperator data and sensor data, a machine learning algorithm to predict future instructions for one or more robots to complete the first task. 
   
     
     
         13 . The system of  claim 12 , further comprising the robot, wherein the robot is configured to:
 capture the sensor data; and   transmit the sensor data to an operator system.   
     
     
         14 . The system of  claim 13 , wherein sensor data comprises an image including one or more objects to be manipulated by the robot to complete the first task. 
     
     
         15 . The system of  claim 13 , further comprising the operator system, wherein the operator system is configured to generate the teleoperator data. 
     
     
         16 . The system of  claim 15 , wherein the robot is further configured to perform the instructions after receiving the teleoperator data from the operator system. 
     
     
         17 . The system of  claim 12 , wherein the machine learning algorithm is a deep learning model and/or a convolutional neural network. 
     
     
         18 . The system of  claim 12 , wherein training the machine learning algorithm comprises:
 calculating a difference between the predicted future instructions and the received instructions; and   adjusting, based on the difference, one or more coefficients of the machine learning algorithm to reduce the difference.   
     
     
         19 . The system of  claim 12 , wherein the imitation learning engine is further configured to receive an identification of the first task and to train the machine learning algorithm to predict the future instructions using the received identification of the first task. 
     
     
         20 . The system of  claim 12 , wherein the sensor data comprises an image of one or more objects to be manipulated by the robot to complete the first task, and wherein the imitation learning engine is configured to train the machine learning algorithm to predict the future instructions using the image of the one or more objects.