Distributed Proximal Policy Optimization for Contention-Based Spectrum Access

TL;DR

This paper introduces a distributed policy gradient method based on Proximal Policy Optimization for contention-based spectrum access, enabling decentralized decision-making to improve fairness and throughput in wireless networks.

Contribution

It presents a novel distributed implementation of PPO tailored for spectrum access, combining spectrum sensing with reinforcement learning for autonomous medium access decisions.

Findings

Significantly higher proportional fairness reward compared to adaptive energy detection thresholds

Improved sum throughput in network scenarios

Enhanced maximum user throughput

Abstract

The increasing number of wireless devices operating in unlicensed spectrum motivates the development of intelligent adaptive approaches to spectrum access that go beyond traditional carrier sensing. We develop a novel distributed implementation of a policy gradient method known as Proximal Policy Optimization modelled on a two stage Markov decision process that enables such an intelligent approach, and still achieves decentralized contention-based medium access. In each time slot, a base station (BS) uses information from spectrum sensing and reception quality to autonomously decide whether or not to transmit on a given resource, with the goal of maximizing proportional fairness network-wide. Empirically, we find the proportional fairness reward accumulated by the policy gradient approach to be significantly higher than even a genie-aided adaptive energy detection threshold. This is further validated by the improved sum and maximum user throughputs achieved by our approach.