Robot control model learning method for reducing frequency of robot intervention behavior

Abstract

A robot control model learning device ( 10 ) performs, by using state information indicating the state of a robot which autonomously travels to a destination in a dynamic environment as an input, reinforcement learning to obtain a robot control model for selecting and outputting a behavior in accordance with the state of the robot from among a plurality of behaviors including an intervention behavior for intervening in the environment, while using the number of times the intervention behavior has been performed as a minus reward.

The invention claimed is: 
     
         1 . A robot control model learning method, comprising, by a computer:
 subjecting a robot control model to reinforcement learning, the robot control model being input with state information expressing a state of a robot that travels autonomously to a destination in a dynamic environment, and the robot control model selecting and outputting a behavior corresponding to the state of the robot from among a plurality of behaviors, wherein:
 the plurality of behaviors include an intervention behavior of intervening in the dynamic environment, 
 the intervention behavior is a non-moving behavior of the robot to notify someone of an existence of the robot in order for the robot to travel to the destination without stopping, 
 the reinforcement learning is performed on the robot control model using a reward function, 
 the reward function includes a weighted combination of an environment reward and an influence reward, 
 the environment reward takes a first negative value when the robot collides with another object, 
 the influence reward takes a second negative value when the robot executes the intervention behavior, and 
 the reinforcement learning comprises:
 learning a deep neural network representing a state value function, 
 determining the state of the robot based on: the reward function, a discount factor, an increase in time in one step, and the state value function, and 
 updating the state value function by updating parameters of the deep neural network. 
 
   
     
     
         2 . The robot control model learning method of  claim 1 , wherein:
 the plurality of behaviors include at least one of a moving direction of the robot, a moving velocity of the robot, or the intervention behavior, and   the reward function is given such that at least one of an arrival time until the robot reaches the destination or a number of times of intervention decreases.   
     
     
         3 . The robot control model learning method of  claim 1 , wherein:
 the plurality of behaviors include an avoidance behavior, which is a moving behavior of the robot to avoid collision with another object when travelling to the destination, and   the reward function is given such that a number of times of avoidance in which the collision is avoided decreases.   
     
     
         4 . The robot control model learning method of  claim 1 , wherein:
 the discount factor is set to a value between zero and one, and   a reward that is obtained at a later time is to be discounted more than a reward that is obtained at an earlier time, based on the discount factor and the increase in time in one step.   
     
     
         5 . A robot control model learning device, comprising:
 a learning section that includes a processor and a memory and subjects a robot control model to reinforcement learning, the robot control model being configured to receive state information expressing a state of a robot that travels autonomously to a destination in a dynamic environment, and to select and output a behavior corresponding to the state of the robot from among a plurality of behaviors, wherein:
 the plurality of behaviors include an intervention behavior of intervening in the dynamic environment, 
 the intervention behavior is a non-moving behavior of the robot to notify someone of an existence of the robot in order for the robot to travel to the destination without stopping, 
 the reinforcement learning is performed on the robot control model using a reward function, 
 the reward function includes a weighted combination of an environment reward and an influence reward, 
 the environment reward takes a first negative value when the robot collides with another object, 
 the influence reward takes a second negative value when the robot executes the intervention behavior, and 
 the reinforcement learning comprises:
 learning a deep neural network representing a state value function, 
 determining the state of the robot based on: the reward function, a discount factor, an increase in time in one step, and the state value function, and 
 updating the state value function by updating parameters of the deep neural network. 
 
   
     
     
         6 . A non-transitory recording medium storing a robot control model learning program that is executable by a computer to perform processing, the processing comprising:
 subjecting a robot control model to reinforcement learning, the robot control model being input with state information expressing a state of a robot that travels autonomously to a destination in a dynamic environment, and the robot control model selecting and outputting a behavior corresponding to the state of the robot from among a plurality of behaviors, wherein:
 the plurality of behaviors include an intervention behavior of intervening in the dynamic environment, 
 the intervention behavior is a non-moving behavior of the robot to notify someone of an existence of the robot in order for the robot to travel to the destination without stopping, 
 the reinforcement learning is performed on the robot control model using a reward function, 
 the reward function includes a weighted combination of an environment reward and an influence reward, 
 the environment reward takes a first negative value when the robot collides with another object, 
 the influence reward takes a second negative value when the robot executes the intervention behavior, and 
 the reinforcement learning comprises:
 learning a deep neural network representing a state value function, 
 determining the state of the robot based on: the reward function, a discount factor, an increase in time in one step, and the state value function, and 
 updating the state value function by updating parameters of the deep neural network. 
 
   
     
     
         7 . A robot control method, according to which a computer performs processing, the processing comprising:
 acquiring state information expressing a state of a robot that travels autonomously to a destination in a dynamic environment; and   effecting control such that the robot moves to the destination on the basis of the state information and on the basis of the robot control model learned by the robot control model learning method of  claim 1 .   
     
     
         8 . A robot control device, comprising:
 an acquisition section that is implemented on a processor and a memory and acquires state information expressing a state of a robot that travels autonomously to a destination in a dynamic environment; and   a control section that is implemented on a processor and a memory and effects control such that the robot moves to the destination on the basis of the state information and on the basis of the robot control model learned by the robot control model learning device of  claim 5 .   
     
     
         9 . A non-transitory recording medium storing a robot control program that is executable by a computer to perform processing, the processing comprising:
 acquiring state information expressing a state of a robot that travels autonomously to a destination in a dynamic environment; and   effecting control such that the robot moves to the destination on the basis of the state information and on the basis of the robot control model learned by the robot control model learning method of  claim 1 .   
     
     
         10 . A robot, comprising:
 an acquisition section that is implemented on a processor and a memory and acquires state information expressing a state of the robot, which travels autonomously to a destination in a dynamic environment;   an autonomous traveling section that includes a motor and causes the robot to travel autonomously; and   a robot control device that includes a control section implemented on a processor and a memory and configured to effect control such that the robot moves to the destination on the basis of the state information and on the basis of the robot control model learned by the robot control model learning device of  claim 5 .   
     
     
         11 . A robot control method, according to which a computer performs processing, the processing comprising:
 acquiring state information expressing a state of a robot that travels autonomously to a destination in a dynamic environment; and   effecting control such that the robot moves to the destination on the basis of the state information and on the basis of the robot control model learned by the robot control model learning method of  claim 2 .   
     
     
         12 . A robot control method, according to which a computer performs processing, the processing comprising:
 acquiring state information expressing a state of a robot that travels autonomously to a destination in a dynamic environment; and   effecting control such that the robot moves to the destination on the basis of the state information and on the basis of the robot control model learned by the robot control model learning method of  claim 3 .   
     
     
         13 . A robot control method, according to which a computer performs processing, the processing comprising:
 acquiring state information expressing a state of a robot that travels autonomously to a destination in a dynamic environment; and   effecting control such that the robot moves to the destination on the basis of the state information and on the basis of the robot control model learned by the robot control model learning method of  claim 1 .   
     
     
         14 . A non-transitory recording medium storing a robot control program that is executable by a computer to perform processing, the processing comprising:
 acquiring state information expressing a state of a robot that travels autonomously to a destination in a dynamic environment; and   effecting control such that the robot moves to the destination on the basis of the state information and on the basis of the robot control model learned by the robot control model learning method of  claim 2 .   
     
     
         15 . A non-transitory recording medium storing a robot control program that is executable by a computer to perform processing, the processing comprising:
 acquiring state information expressing a state of a robot that travels autonomously to a destination in a dynamic environment; and   effecting control such that the robot moves to the destination on the basis of the state information and on the basis of the robot control model learned by the robot control model learning method of  claim 3 .   
     
     
         16 . A non-transitory recording medium storing a robot control program that is executable by a computer to perform processing, the processing comprising:
 acquiring state information expressing a state of a robot that travels autonomously to a destination in a dynamic environment; and   effecting control such that the robot moves to the destination on the basis of the state information and on the basis of the robot control model learned by the robot control model learning method of  claim 1 .   
     
     
         17 . The robot control model learning device of  claim 5 , wherein:
 the plurality of behaviors include at least one of a moving direction of the robot, a moving velocity of the robot, or the intervention behavior, and   the reward function is given such that at least one of an arrival time until the robot reaches the destination or a number of times of intervention decreases.   
     
     
         18 . The robot control model learning method of  claim 5 , wherein:
 the plurality of behaviors include an avoidance behavior, which is a moving behavior of the robot to avoid a collision with another object when travelling to the destination, and   the reward function is given such that a number of times of avoidance in which the collision is avoided decreases.   
     
     
         19 . The non-transitory recording medium of  claim 6 , wherein:
 the plurality of behaviors include at least one of a moving direction of the robot, a moving velocity of the robot, or the intervention behavior, and   the reward function is given such that at least one of an arrival time until the robot reaches the destination or a number of times of intervention decreases.   
     
     
         20 . The non-transitory recording medium of  claim 6 , wherein:
 the plurality of behaviors include an avoidance behavior, which is a moving behavior of the robot to avoid a collision with another object when travelling to the destination, and   the reward function is given such that a number of times of avoidance in which the collision is avoided decreases.