Deterministic and Stochastic Analysis of Deep Reinforcement Learning for Low Dimensional Sensing-based Navigation of Mobile Robots

TL;DR

This paper compares deterministic and stochastic deep reinforcement learning methods, DDPG and SAC, for mapless navigation of mobile robots with high-dimensional sensing, analyzing how neural network architecture impacts their performance.

Contribution

It provides a comparative analysis of DDPG and SAC in high-dimensional navigation tasks, highlighting the influence of neural network architecture on learning outcomes.

Findings

SAC performs better with deeper neural network architectures.

DDPG is more effective with shallower architectures.

Quantitative results show differences in navigation time and distance.

Abstract

Deterministic and Stochastic techniques in Deep Reinforcement Learning (Deep-RL) have become a promising solution to improve motion control and the decision-making tasks for a wide variety of robots. Previous works showed that these Deep-RL algorithms can be applied to perform mapless navigation of mobile robots in general. However, they tend to use simple sensing strategies since it has been shown that they perform poorly with a high dimensional state spaces, such as the ones yielded from image-based sensing. This paper presents a comparative analysis of two Deep-RL techniques - Deep Deterministic Policy Gradients (DDPG) and Soft Actor-Critic (SAC) - when performing tasks of mapless navigation for mobile robots. We aim to contribute by showing how the neural network architecture influences the learning itself, presenting quantitative results based on the time and distance of navigation of aerial mobile robots for each approach. Overall, our analysis of six distinct architectures highlights that the stochastic approach (SAC) better suits with deeper architectures, while the opposite happens with the deterministic approach (DDPG).