Perturbation-mitigated USV Navigation with Distributionally Robust Reinforcement Learning

TL;DR

This paper introduces DRIQN, a novel distributionally robust reinforcement learning method that enhances USV navigation safety and efficiency in noisy, unpredictable marine environments by optimizing worst-case performance.

Contribution

The paper proposes DRIQN, integrating DRO with implicit quantile networks and explicit subgroup modeling to improve robustness against heterogeneous environmental noise in USV navigation.

Findings

DRIQN achieves +13.51% success rate over state-of-the-art.

Reduces collision rate by -12.28%.

Improves time and energy efficiency significantly.

Abstract

The robustness of Unmanned Surface Vehicles (USV) is crucial when facing unknown and complex marine environments, especially when heteroscedastic observational noise poses significant challenges to sensor-based navigation tasks. Recently, Distributional Reinforcement Learning (DistRL) has shown promising results in some challenging autonomous navigation tasks without prior environmental information. However, these methods overlook situations where noise patterns vary across different environmental conditions, hindering safe navigation and disrupting the learning of value functions. To address the problem, we propose DRIQN to integrate Distributionally Robust Optimization (DRO) with implicit quantile networks to optimize worst-case performance under natural environmental conditions. Leveraging explicit subgroup modeling in the replay buffer, DRIQN incorporates heterogeneous noise sources and target robustness-critical scenarios. Experimental results based on the risk-sensitive environment demonstrate that DRIQN significantly outperforms state-of-the-art methods, achieving +13.51\% success rate, -12.28\% collision rate and +35.46\% for time saving, +27.99\% for energy saving, compared with the runner-up.