Heterogeneous Multi-Agent Reinforcement Learning for Distributed Channel Access in WLANs

TL;DR

This paper introduces QPMIX, a heterogeneous multi-agent reinforcement learning framework for distributed channel access in WLANs, improving throughput and fairness through collaborative learning among diverse agents.

Contribution

It proposes a novel heterogeneous MARL training framework with convergence proof, enabling collaboration among different RL algorithms for enhanced WLAN performance.

Findings

QPMIX outperforms CSMA/CA in throughput, delay, and collision metrics.

The method is robust in various traffic scenarios.

It ensures fairness and cooperation among heterogeneous agents.

Abstract

This paper investigates the use of multi-agent reinforcement learning (MARL) to address distributed channel access in wireless local area networks. In particular, we consider the challenging yet more practical case where the agents heterogeneously adopt value-based or policy-based reinforcement learning algorithms to train the model. We propose a heterogeneous MARL training framework, named QPMIX, which adopts a centralized training with distributed execution paradigm to enable heterogeneous agents to collaborate. Moreover, we theoretically prove the convergence of the proposed heterogeneous MARL method when using the linear value function approximation. Our method maximizes the network throughput and ensures fairness among stations, therefore, enhancing the overall network performance. Simulation results demonstrate that the proposed QPMIX algorithm improves throughput, mean delay, delay jitter, and collision rates compared with conventional carrier-sense multiple access with collision avoidance (CSMA/CA) mechanism in the saturated traffic scenario. Furthermore, the QPMIX algorithm is robust in unsaturated and delay-sensitive traffic scenarios. It coexists well with the conventional CSMA/CA mechanism and promotes cooperation among heterogeneous agents.