Robot dynamic obstacle avoidance method based on multimodal spiking neural network

Abstract

The present invention provides a robot dynamic obstacle avoidance method based on a multimodal spiking neural network. The present invention realizes a robot obstacle avoidance method in a dynamic environment by fusing laser radar data and processed event camera data and combining with the intrinsic learnable threshold of the spiking neural network for a scenario comprising dynamic obstacles. It solves the difficulty of failure of obstacle avoidance due to the difficulty in perceiving the dynamic obstacles in the obstacle avoidance task of a robot. The present invention helps the robot to fully perceive the static information and the dynamic information of the environment, uses the learnable threshold mechanism of the spiking neural network for efficient reinforcement learning training and decision making, and realizes autonomous navigation and obstacle avoidance in the dynamic environment. An event data enhanced model is combined to better adapt to the dynamic environment for obstacle avoidance.

The invention claimed is: 
     
       1. A robot dynamic obstacle avoidance method based on a multimodal spiking neural network, comprising the following steps:
 step 1, carrying a robot simulation model; 
 carrying a two-dimensional laser radar and an event camera simultaneously by a robot for perceiving an environment and acquiring laser radar data and event data; 
 step 2, building a hybrid spiking variational autoencoder module to generate event camera data; 
 encoding the original (x, x) event data with sparse features by the hybrid spiking variational autoencoder module and simplifying into (1, x/2) one-dimensional vector event camera data with highly concentrated features; and acquiring the event data from an event camera carried by the robot to form a dataset which is inputted to the hybrid spiking variational autoencoder module for generating a low-dimensional latent vector as the event camera data inputted by a subsequent population coding module; 
 the hybrid spiking variational autoencoder module comprises a spiking variational autoencoder and a decoder; the spiking variational autoencoder comprises 4 layers of convolutional spiking neural networks, and each layer of convolutional spiking neural network is composed of LIF (Leaky Integrate-and-Fire) neurons; the spiking variational autoencoder records the states of all the LIF neurons in a path process of data interaction with the robot at each moment and transmits the states to a next moment for learning the weight of the spiking variational autoencoder; the decoder comprises 4 layers of deconvolutional artificial neural networks; the spiking variational autoencoder is responsible for learning (x, x)-dimensional event data features and storing into an x/2-dimensional latent vector; the decoder is used for reversely verifying the validity of the spiking variational autoencoder, and reconstructing the value of the latent vector into original event data by taking a conventional UAE (variational autoencoder) loss function as an optimization objective; and when the decoder can reconstruct the original event data, it represents that the training of the spiking variational autoencoder is completed; 
 step 3, encoding multimodal data into spiking sequence data by population coding and Poisson coding; 
 connecting the event camera data and the laser radar data in series into multimodal data; converting the multimodal data into a stimulation strength value by the population coding module, and generating, by Poisson coding, the spiking sequence data from the stimulation strength value for direct input into a subsequent middle fusion decision module; 
 the population coding module comprises 10 LIF neurons for making up for the inadequacy of single LIF neuron coding and reducing information loss when the multimodal data is converted to the spiking sequence data; 
 step 4, constructing the middle fusion decision module which comprises a middle fusion module and a control decision module; inputting the spiking sequence data obtained in step 3 into the middle fusion decision module to output the motion decision of the robot; 
 step 4.1, aligning, by the middle fusion module, the event camera spiking sequence data and the laser radar spiking sequence data into two (1,c) one-dimensional vectors through the LIF neurons composed of two fully connected layers, and connecting the two one-dimensional vectors directly to form fused feature data; adding the middle fusion module into a learnable threshold mechanism; calculating the learnable threshold by a tanh (x) function; when the middle fusion module conducts back propagation, updating the network weight and the learnable threshold of the middle fusion module; controlling, by the learnable threshold, the firing frequency of information transmitted by the LIF neurons, and according to the update of the threshold, conducting adaptive fusion of the event camera data and the laser radar data at different firing frequencies to obtain feature data; 
 step 4.2, the control decision module comprises four fully connected layers built by the spiking neural network; the fully connected layers are composed of the LIF neurons; embedding the control decision module into a deep reinforcement learning framework DDPG, replacing an actor network of the existing deep reinforcement learning framework DDPG by the spiking neural network to make decisions in the form of spiking, conducting autonomous trial and error learning and determining the threshold of the middle fusion module until optimal feature data is confirmed; 
 the input of the control decision module is the feature data fused by the middle fusion module; making action decisions through the four fully connected layers; taking a mean value added by the output values of the control decision module on all time steps as a value that represents the values of the left and right wheel speeds of the robot; and then converting into the action output of the linear and angular velocities through the dynamics of the robot to conduct autonomous perception and decision; 
 adding all the LIF neurons in the control decision module into the learnable threshold mechanism; calculating the learnable threshold by the tanh (x) function; and when the control decision module conducts back propagation, updating the network weight and the learnable threshold of the control decision module so that the threshold of each layer of LIF neurons is maintained at a different level. 
 
     
     
       2. The robot dynamic obstacle avoidance method based on the multimodal spiking neural network according to  claim 1 , wherein a URDF model of a TurtleBot-ROS robot is selected by the robot as an experimental robot; and the x is 128. 
     
     
       3. The robot dynamic obstacle avoidance method based on the multimodal spiking neural network according to  claim 1 , wherein the laser radar data is an 18-dimensional vector, the event camera data is a 64-dimensional vector, and the robot speed information and the robot distance information are both 3-dimensional vectors. 
     
     
       4. The robot dynamic obstacle avoidance method based on the multimodal spiking neural network according to  claim 2 , wherein the laser radar data is an 18-dimensional vector, the event camera data is a 64-dimensional vector, and the robot speed information and the robot distance information are both 3-dimensional vectors.