Deep Reinforcement Learning for Simultaneous Sensing and Channel Access in Cognitive Networks

TL;DR

This paper introduces a novel deep reinforcement learning algorithm, DDQSA, for dynamic spectrum access in cognitive networks, enabling secondary users to learn optimal sensing and access policies from partial observations.

Contribution

It is the first to apply deep Q-learning to jointly optimize sensing and access policies in DSA, with theoretical analysis of optimal policies in cyclic user models.

Findings

DDQSA outperforms existing methods in throughput.

Theoretical validation of learned policies.

Explicit optimal policy derivation for cyclic models.

Abstract

We consider the problem of dynamic spectrum access (DSA) in cognitive wireless networks, where only partial observations are available to the users due to narrowband sensing and transmissions. The cognitive network consists of primary users (PUs) and a secondary user (SU), which operate in a time duplexing regime. The traffic pattern for each PU is assumed to be unknown to the SU and is modeled as a finite-memory Markov chain. Since observations are partial, then both channel sensing and access actions affect the throughput. The objective is to maximize the SU's long-term throughput. To achieve this goal, we develop a novel algorithm that learns both access and sensing policies via deep Q-learning, dubbed Double Deep Q-network for Sensing and Access (DDQSA). To the best of our knowledge, this is the first paper that solves both sensing and access policies for DSA via deep Q-learning. Second, we analyze the optimal policy theoretically to validate the performance of DDQSA. Although the general DSA problem is P-SPACE hard, we derive the optimal policy explicitly for a common model of a cyclic user dynamics. Our results show that DDQSA learns a policy that implements both sensing and channel access, and significantly outperforms existing approaches.