A Deep Reinforcement Learning Framework for Contention-Based Spectrum Sharing

TL;DR

This paper introduces a decentralized deep reinforcement learning framework for spectrum sharing that optimizes network-wide fairness and adapts to environment dynamics in unlicensed spectrum.

Contribution

It formulates a novel decentralized POMDP with a unique reward structure and develops a distributed RL approach for contention-based spectrum access.

Findings

Achieves competitive proportional fairness performance.

Robust to channel fading and small contention windows.

Outperforms traditional adaptive energy detection methods.

Abstract

The increasing number of wireless devices operating in unlicensed spectrum motivates the development of intelligent adaptive approaches to spectrum access. We consider decentralized contention-based medium access for base stations (BSs) operating on unlicensed shared spectrum, where each BS autonomously decides whether or not to transmit on a given resource. The contention decision attempts to maximize not its own downlink throughput, but rather a network-wide objective. We formulate this problem as a decentralized partially observable Markov decision process with a novel reward structure that provides long term proportional fairness in terms of throughput. We then introduce a two-stage Markov decision process in each time slot that uses information from spectrum sensing and reception quality to make a medium access decision. Finally, we incorporate these features into a distributed reinforcement learning framework for contention-based spectrum access. Our formulation provides decentralized inference, online adaptability and also caters to partial observability of the environment through recurrent Q-learning. Empirically, we find its maximization of the proportional fairness metric to be competitive with a genie-aided adaptive energy detection threshold, while being robust to channel fading and small contention windows.