Consensus-based Decentralized Multi-agent Reinforcement Learning for Random Access Network Optimization

TL;DR

This paper introduces a fully decentralized multi-agent reinforcement learning algorithm for random access network optimization, reducing communication overhead and providing theoretical convergence guarantees, leading to improved network performance.

Contribution

It presents a novel consensus-based decentralized MARL approach with local reward exchange and proven global convergence for RA MAC protocol optimization.

Findings

Significant performance improvements over baseline methods.

Reduced communication overhead through local reward exchange.

Theoretical proof of convergence for the proposed algorithm.

Abstract

With wireless devices increasingly forming a unified smart network for seamless, user-friendly operations, random access (RA) medium access control (MAC) design is considered a key solution for handling unpredictable data traffic from multiple terminals. However, it remains challenging to design an effective RA-based MAC protocol to minimize collisions and ensure transmission fairness across the devices. While existing multi-agent reinforcement learning (MARL) approaches with centralized training and decentralized execution (CTDE) have been proposed to optimize RA performance, their reliance on centralized training and the significant overhead required for information collection can make real-world applications unrealistic. In this work, we adopt a fully decentralized MARL architecture, where policy learning does not rely on centralized tasks but leverages consensus-based information exchanges across devices. We design our MARL algorithm over an actor-critic (AC) network and propose exchanging only local rewards to minimize communication overhead. Furthermore, we provide a theoretical proof of global convergence for our approach. Numerical experiments show that our proposed MARL algorithm can significantly improve RA network performance compared to other baselines.