Fast Policy Learning for 6-DOF Position Control of Underwater Vehicles

TL;DR

This paper presents a GPU-accelerated reinforcement learning pipeline for rapid 6-DOF position control of underwater vehicles, enabling fast training and effective real-world deployment with zero-shot transfer.

Contribution

It introduces a novel JAX-based, GPU-accelerated RL training method that drastically reduces training time for underwater vehicle control policies.

Findings

Achieved training times under two minutes using GPU acceleration.

Demonstrated robust trajectory tracking and disturbance rejection in real experiments.

Policies transferred zero-shot from simulation to real underwater vehicles.

Abstract

Autonomous Underwater Vehicles (AUVs) require reliable six-degree-of-freedom (6-DOF) position control to operate effectively in complex and dynamic marine environments. Traditional controllers are effective under nominal conditions but exhibit degraded performance when faced with unmodeled dynamics or environmental disturbances. Reinforcement learning (RL) provides a powerful alternative but training is typically slow and sim-to-real transfer remains challenging. This work introduces a GPU accelerated RL training pipeline built in JAX and MuJoCo-XLA (MJX). By jointly JIT-compiling large-scale parallel physics simulation and learning updates, we achieve training times of under two minutes. Through systematic evaluation of multiple RL algorithms, we show robust 6-DOF trajectory tracking and effective disturbance rejection in real underwater experiments, with policies transferred zero-shot from simulation.