Systems and methods for using reinforcement learning agents to track targets based on a reward including an information metric reward

Abstract

An estimated target position is generated based on agent data received from an agent sensor system of a reinforcement learning (RL) agent and teammate data received from a plurality of teammate RL agents. An information metric reward is generated based on a confidence level associated with the estimated target position. The confidence level is based on the estimated target position and historical estimated target positions. A distance metric reward is generated based on an agent position and the estimated target position. A combined reward is generated based on the information metric reward and the distance metric reward. A movement action is generated for the RL agent based on the agent position, the estimated target position, and the combined reward in accordance with a multi-agent reinforcement learning (MARL) algorithm.

What is claimed is: 
     
       1. A reinforcement learning (RL) agent comprising:
 a sensor system; 
 a communication system; and 
 a target tracking system comprising:
 at least one processor; and 
 at least one memory communicatively coupled to the at least one processor, the at least one memory comprising instructions that upon execution by the at least one processor, cause the at least one processor to:
 receive agent data comprising an agent position and sensor target data from the sensor system; 
 receive teammate data from a plurality of teammate RL agents via the communication system, the teammate data for each of the plurality of teammate RL agents comprising a teammate position and teammate sensor target data; 
 generate an estimated target position of a target based on the agent data and the teammate data; 
 generate an information metric reward based on a confidence level associated with the estimated target position, the confidence level being based on the estimated target position and historical estimated target positions for the target; 
 generate a distance metric reward based on the agent position and the estimated target position; 
 generate a combined reward based on the information metric reward and the distance metric reward; and 
 generate a movement action for the RL agent based on the agent position, the estimated target position, and the combined reward in accordance with a multi-agent reinforcement learning (MARL) algorithm. 
 
 
 
     
     
       2. The RL agent of  claim 1 , wherein the RL agent is an unmanned aerial vehicle (UAV). 
     
     
       3. The RL agent of  claim 1 , wherein the at least one memory comprises instructions that upon execution by the at least one processor, cause the at least one processor to:
 receive the agent data and the teammate data at a Kalman filter; and 
 generate, by the Kalman filter, the estimated target position based on the agent data and the teammate data. 
 
     
     
       4. The RL agent of  claim 1 , wherein the at least one memory comprises instructions that upon execution by the at least one processor, cause the at least one processor to:
 calculate Fisher Information for the estimated target position, the Fisher Information being operable to provide the confidence level associated with the estimated target position using a scalar value; and 
 generate the information metric reward based on the scalar value. 
 
     
     
       5. The RL agent of  claim 1 , wherein the at least one memory comprises instructions that upon execution by the at least one processor, cause the at least one processor to generate the distance metric reward based on a measured distance between the estimated target position and the agent position. 
     
     
       6. The RL agent of  claim 1 , wherein the MARL algorithm comprises a Deep Q learning (DQN) algorithm. 
     
     
       7. The RL agent of  claim 1 , wherein the MARL algorithm comprises a Deep Deterministic Policy Gradient (DDPG) algorithm. 
     
     
       8. The RL agent of  claim 1 , wherein the at least one memory comprises instructions that upon execution by the at least one processor, cause the at least one processor to generate the movement action for the RL agent, the movement action being one of a forward movement action, a backward movement action, a left movement action, and a right movement action. 
     
     
       9. The RL agent of  claim 1 , wherein the at least one memory comprises instructions that upon execution by the at least one processor, cause the at least one processor to:
 apply a first weight to the information metric reward to generate a weighted information metric reward; 
 apply a second weight to the distance metric reward to generate a weighted distance metric reward, the first weight being different from the second weight; and 
 generate the combined reward based on a sum of the weighted information metric reward and the weighted distance metric reward. 
 
     
     
       10. The RL agent of  claim 1 , wherein the at least one memory comprises instructions that upon execution by the at least one processor, cause the at least one processor to:
 enable the RL agent to track the target within a finite environment; and 
 receive the teammate data from the plurality of teammate RL agents, the plurality of teammate RL agents being configured to track the target within the finite environment. 
 
     
     
       11. A method of tracking a target by a reinforcement learning (RL) agent comprising:
 receiving agent data comprising an agent position and sensor target data from a sensor system of the RL agent; 
 receiving teammate data from a plurality of teammate RL agents via a communication system of the RL agent, the teammate data for each of the plurality of teammate agents comprising a teammate agent position and teammate sensor target data; 
 generating an estimated target position of the target based on the agent data and the teammate data; 
 generating an information metric reward based on a confidence level associated with the estimated target position, the confidence level being based on the estimated target position and historical estimated target positions for the target; 
 generating a distance metric reward based on the agent position and the estimated target position; 
 generating a combined reward based on the information metric reward and the distance metric reward; and 
 generating a movement action for the RL agent based on the agent position, the estimated target position, and the combined reward in accordance with a multi-agent reinforcement learning (MARL) algorithm. 
 
     
     
       12. The method of  claim 11 , wherein the agent comprises an unmanned aerial vehicle (UAV). 
     
     
       13. The method of  claim 11 , further comprising:
 receiving the agent data and the teammate data received at a Kalman filter; and 
 generating, by the Kaman filter, the estimated target position based on the agent data and the teammate data. 
 
     
     
       14. The method of  claim 11 , further comprising:
 calculating Fisher Information for the estimated target position, the Fisher Information being operable to provide the confidence level associated with the estimated target position using a scalar value; and 
 generating the information metric reward based on the scalar value. 
 
     
     
       15. The method of  claim 11 , further comprising generating the distance metric reward based on a measured distance between the estimated target position and the agent position. 
     
     
       16. The method of  claim 11 , wherein the MARL algorithm comprises a Deep Q learning (DQN) algorithm. 
     
     
       17. The method of  claim 11 , wherein the MARL algorithm comprises a Deep Deterministic Policy Gradient (DDPG) algorithm. 
     
     
       18. The method of  claim 11 , further comprising generating the movement action for the RL agent, the movement action being one of a forward movement action, a backward movement action, a left movement action, and a right movement action. 
     
     
       19. The method of  claim 11 , further comprising:
 applying a first weight to the information metric reward to generate a weighted information metric reward; 
 applying a second weight to the distance metric reward to generate a weighted distance metric reward, the first weight being different from the second weight; and 
 generating the combined reward based on a sum of the weighted information metric reward and the weighted distance metric reward. 
 
     
     
       20. The method of  claim 11 , further comprising:
 enabling the RL agent to track the target within a finite environment; and 
 receiving teammate data from the plurality of teammate RL agents, the plurality of teammate RL agents being configured to track the target within the finite environment.