Reinforcement Learning Random Access for Delay-Constrained Heterogeneous Wireless Networks: A Two-User Case

TL;DR

This paper introduces a reinforcement learning-based random access scheme for a two-user delay-constrained wireless network, achieving near-optimal performance and outperforming existing methods by optimizing spectrum sharing without full user information.

Contribution

It proposes a novel RL-based random access scheme, TSRA, for delay-constrained heterogeneous networks, with a model-based upper bound and extensive simulation validation.

Findings

TSRA achieves performance close to the theoretical upper bound.

TSRA outperforms existing baseline schemes in delay-constrained scenarios.

The approach effectively adapts to unknown user behaviors in spectrum sharing.

Abstract

In this paper, we investigate the random access problem for a delay-constrained heterogeneous wireless network. As a first attempt to study this new problem, we consider a network with two users who deliver delay-constrained traffic to an access point (AP) via a common unreliable collision wireless channel. We assume that one user (called user 1) adopts ALOHA and we optimize the random access scheme of the other user (called user 2). The most intriguing part of this problem is that user 2 does not know the information of user 1 but needs to maximize the system timely throughput. Such a paradigm of collaboratively sharing spectrum is envisioned by DARPA to better dynamically match the supply and demand in the future [1], [2]. We first propose a Markov Decision Process (MDP) formulation to derive a modelbased upper bound, which can quantify the performance gap of any designed schemes. We then utilize reinforcement learning (RL) to design an R-learning-based [3]-[5] random access scheme, called TSRA. We finally carry out extensive simulations to show that TSRA achieves close-to-upper-bound performance and better performance than the existing baseline DLMA [6], which is our counterpart scheme for delay-unconstrained heterogeneous wireless network. All source code is publicly available in https://github.com/DanzhouWu/TSRA.