Double Critic Deep Reinforcement Learning for Mapless 3D Navigation of Unmanned Aerial Vehicles

TL;DR

This paper introduces a deep reinforcement learning system using double critic models and RNNs for UAV mapless 3D navigation, outperforming previous methods with sparse range data.

Contribution

The paper proposes a novel deep RL framework with double critic models and RNNs for UAV navigation using sparse sensor data, improving over existing approaches.

Findings

Double critic models outperform DDPG and BUG2 algorithms.

RNN-based deep RL models outperform previous navigation structures.

Sparse range data is sufficient for effective UAV navigation.

Abstract

This paper presents a novel deep reinforcement learning-based system for 3D mapless navigation for Unmanned Aerial Vehicles (UAVs). Instead of using a image-based sensing approach, we propose a simple learning system that uses only a few sparse range data from a distance sensor to train a learning agent. We based our approaches on two state-of-art double critic Deep-RL models: Twin Delayed Deep Deterministic Policy Gradient (TD3) and Soft Actor-Critic (SAC). We show that our two approaches manage to outperform an approach based on the Deep Deterministic Policy Gradient (DDPG) technique and the BUG2 algorithm. Also, our new Deep-RL structure based on Recurrent Neural Networks (RNNs) outperforms the current structure used to perform mapless navigation of mobile robots. Overall, we conclude that Deep-RL approaches based on double critic with Recurrent Neural Networks (RNNs) are better suited to perform mapless navigation and obstacle avoidance of UAVs.