A Multi-Agent Deep Reinforcement Learning based Spectrum Allocation Framework for D2D Communications

TL;DR

This paper introduces a distributed multi-agent deep reinforcement learning framework called NAAC for spectrum allocation in D2D communications, reducing interference and improving system performance without extensive signaling.

Contribution

It proposes a novel neighbor-agent actor critic framework that enables distributed spectrum allocation using historical data for training, enhancing efficiency and cooperation.

Findings

Reduces outage probability of cellular links

Improves sum rate of D2D links

Demonstrates good convergence in simulations

Abstract

Device-to-device (D2D) communication has been recognized as a promising technique to improve spectrum efficiency. However, D2D transmission as an underlay causes severe interference, which imposes a technical challenge to spectrum allocation. Existing centralized schemes require global information, which can cause serious signaling overhead. While existing distributed solution requires frequent information exchange between users and cannot achieve global optimization. In this paper, a distributed spectrum allocation framework based on multi-agent deep reinforcement learning is proposed, named Neighbor-Agent Actor Critic (NAAC). NAAC uses neighbor users' historical information for centralized training but is executed distributedly without that information, which not only has no signal interaction during execution, but also utilizes cooperation between users to further optimize system performance. The simulation results show that the proposed framework can effectively reduce the outage probability of cellular links, improve the sum rate of D2D links and have good convergence.