Sim-to-reality adaptation for Deep Reinforcement Learning applied to an underwater docking application

TL;DR

This paper presents a sim-to-real adaptation framework for deep reinforcement learning applied to autonomous underwater docking, using a high-fidelity digital twin and PPO algorithm to achieve over 90% success in simulation and validate in real tank tests.

Contribution

It introduces a high-fidelity digital twin environment and a multiprocessing RL framework for efficient training of underwater docking policies with successful sim-to-real transfer.

Findings

Over 90% success rate in simulation

Validated effectiveness in physical tank tests

Emergent behaviors aiding docking process

Abstract

Deep Reinforcement Learning (DRL) offers a robust alternative to traditional control methods for autonomous underwater docking, particularly in adapting to unpredictable environmental conditions. However, bridging the "sim-to-real" gap and managing high training latencies remain significant bottlenecks for practical deployment. This paper presents a systematic approach for autonomous docking using the Girona Autonomous Underwater Vehicle (AUV) by leveraging a high-fidelity digital twin environment. We adapted the Stonefish simulator into a multiprocessing RL framework to significantly accelerate the learning process while incorporating realistic AUV dynamics, collision models, and sensor noise. Using the Proximal Policy Optimization (PPO) algorithm, we developed a 6-DoF control policy trained in a headless environment with randomized starting positions to ensure generalized performance. Our reward structure accounts for distance, orientation, action smoothness, and adaptive collision penalties to facilitate soft docking. Experimental results demonstrate that the agent achieved a success rate of over 90% in simulation. Furthermore, successful validation in a physical test tank confirmed the efficacy of the sim-to-reality adaptation, with the DRL controller exhibiting emergent behaviors such as pitch-based braking and yaw oscillations to assist in mechanical alignment.