Aquatic Navigation: A Challenging Benchmark for Deep Reinforcement Learning

TL;DR

This paper introduces a new aquatic navigation benchmark for deep reinforcement learning, highlighting its challenges and proposing advanced training techniques to improve policy reliability and safety in complex environments.

Contribution

It presents a novel aquatic navigation environment for DRL, demonstrating its difficulty and proposing curriculum learning and hyperparameter tuning to enhance performance.

Findings

State-of-the-art DRL struggles with generalization and safety in aquatic navigation.

Advanced training techniques improve policy reliability.

Benchmark environment and baselines are publicly available.

Abstract

An exciting and promising frontier for Deep Reinforcement Learning (DRL) is its application to real-world robotic systems. While modern DRL approaches achieved remarkable successes in many robotic scenarios (including mobile robotics, surgical assistance, and autonomous driving) unpredictable and non-stationary environments can pose critical challenges to such methods. These features can significantly undermine fundamental requirements for a successful training process, such as the Markovian properties of the transition model. To address this challenge, we propose a new benchmarking environment for aquatic navigation using recent advances in the integration between game engines and DRL. In more detail, we show that our benchmarking environment is problematic even for state-of-the-art DRL approaches that may struggle to generate reliable policies in terms of generalization power and safety. Specifically, we focus on PPO, one of the most widely accepted algorithms, and we propose advanced training techniques (such as curriculum learning and learnable hyperparameters). Our extensive empirical evaluation shows that a well-designed combination of these ingredients can achieve promising results. Our simulation environment and training baselines are freely available to facilitate further research on this open problem and encourage collaboration in the field.