Digital Twin Supervised Reinforcement Learning Framework for Autonomous Underwater Navigation

TL;DR

This paper presents a deep reinforcement learning framework using PPO for autonomous underwater navigation, demonstrating improved obstacle avoidance and transferability from simulation to real-world BlueROV2 operations.

Contribution

It introduces a novel PPO-based deep RL approach with a combined observation space for underwater navigation, validated in simulation and real-world BlueROV2 tests.

Findings

PPO policy outperforms DWA in cluttered environments

Deep RL approach reduces collision rates

Successful transfer from simulation to real-world BlueROV2

Abstract

Autonomous navigation in underwater environments remains a major challenge due to the absence of GPS, degraded visibility, and the presence of submerged obstacles. This article investigates these issues through the case of the BlueROV2, an open platform widely used for scientific experimentation. We propose a deep reinforcement learning approach based on the Proximal Policy Optimization (PPO) algorithm, using an observation space that combines target-oriented navigation information, a virtual occupancy grid, and ray-casting along the boundaries of the operational area. The learned policy is compared against a reference deterministic kinematic planner, the Dynamic Window Approach (DWA), commonly employed as a robust baseline for obstacle avoidance. The evaluation is conducted in a realistic simulation environment and complemented by validation on a physical BlueROV2 supervised by a 3D digital twin of the test site, helping to reduce risks associated with real-world experimentation. The results show that the PPO policy consistently outperforms DWA in highly cluttered environments, notably thanks to better local adaptation and reduced collisions. Finally, the experiments demonstrate the transferability of the learned behavior from simulation to the real world, confirming the relevance of deep RL for autonomous navigation in underwater robotics.