Delay-Robust Deep Reinforcement Learning for Ranging-Free Channel Access under Mobility in Underwater Acoustic Networks

TL;DR

This paper introduces MobiU-MAC, a DRL-based MAC protocol for underwater acoustic networks that is delay-robust and ranging-free, effectively handling mobility and long delays to maximize throughput.

Contribution

It presents CHILL-STER, a novel DRL algorithm optimized for UWANs, with mechanisms for stable learning under delays and mobility, and provides theoretical analysis of DRL's optimality in this context.

Findings

MobiU-MAC outperforms existing DRL-based MAC protocols in underwater environments.

CHILL-STER achieves stable learning with asynchronous delayed rewards.

Theoretical analysis shows DRL attains optimal policies without ranging under long delays.

Abstract

Long propagation delays in underwater acoustic networks (UWANs) cause spatio-temporal uncertainty, constraining channel utilization in medium access control (MAC) protocols. Node mobility within autonomous underwater vehicle scenarios exacerbates these challenges by introducing dynamic propagation delays and varying spatial topologies. We present MobiU-MAC, a deep reinforcement learning (DRL)-based MAC protocol for mobile node access in UWANs that maximizes throughput via autonomous learning. MobiU-MAC incorporates CHILL-STER, a novel DRL algorithm optimized for UWANs that is both ranging-free and delay-robust. CHILL-STER employs a credit horizon-limited $\lambda$-return (CHILL-Return) mechanism to achieve stable learning under asynchronous delayed rewards, while the companion spatio-temporal experience replay (STER) mechanism addresses topological changes arising from node mobility. This work also demonstrates theoretically that DRL attains optimal policy learning equivalent to a standard Markov decision process under long propagation delays without requiring ranging. Performance evaluations indicate that MobiU-MAC outperforms existing DRL-based MAC protocols for UWANs by leveraging the maximum system delay boundary without ranging overhead, supporting the effectiveness of the proposed theory and algorithm in complex underwater dynamic environments.