Learning Random Access Schemes for Massive Machine-Type Communication with MARL

TL;DR

This paper applies multi-agent reinforcement learning techniques to develop grant-free random access schemes for massive machine-type communication, improving throughput and fairness in low-power IoT networks.

Contribution

It introduces MARL-based RA schemes using VDN and QMIX with parameter sharing, and demonstrates their effectiveness in mMTC scenarios with a novel traffic model.

Findings

MARL schemes outperform traditional methods in throughput-fairness trade-off

Including agent identifiers in observations has limited impact on performance

Proposed algorithms adapt well to non-stationary traffic conditions

Abstract

In this paper, we explore various multi-agent reinforcement learning (MARL) techniques to design grant-free random access (RA) schemes for low-complexity, low-power battery operated devices in massive machine-type communication (mMTC) wireless networks. We use value decomposition networks (VDN) and QMIX algorithms with parameter sharing (PS) with centralized training and decentralized execution (CTDE) while maintaining scalability. We then compare the policies learned by VDN, QMIX, and deep recurrent Q-network (DRQN) and explore the impact of including the agent identifiers in the observation vector. We show that the MARL-based RA schemes can achieve a better throughput-fairness trade-off between agents without having to condition on the agent identifiers. We also present a novel correlated traffic model, which is more descriptive of mMTC scenarios, and show that the proposed algorithm can easily adapt to traffic non-stationarities