DRL meets DSA Networks: Convergence Analysis and Its Application to System Design

TL;DR

This paper compares deep reinforcement learning methods for dynamic spectrum access, showing that deep echo state networks require fewer samples and offer better computational efficiency than deep recurrent Q networks.

Contribution

It analytically demonstrates the sample efficiency advantage of DEQN over DRQN and provides a systematic method for hyperparameter tuning in DSA networks.

Findings

DEQN requires fewer training samples than DRQN for convergence.

DEQN outperforms DRQN in computational efficiency.

Performance evaluation confirms DEQN's superiority in DSA applications.

Abstract

In dynamic spectrum access (DSA) networks, secondary users (SUs) need to opportunistically access primary users' (PUs) radio spectrum without causing significant interference. Since the interaction between the SU and the PU systems are limited, deep reinforcement learning (DRL) has been introduced to help SUs to conduct spectrum access. Specifically, deep recurrent Q network (DRQN) has been utilized in DSA networks for SUs to aggregate the information from the recent experiences to make spectrum access decisions. DRQN is notorious for its sample efficiency in the sense that it needs a rather large number of training data samples to tune its parameters which is a computationally demanding task. In our recent work, deep echo state network (DEQN) has been introduced to DSA networks to address the sample efficiency issue of DRQN. In this paper, we analytically show that DEQN comparatively requires less amount of training samples than DRQN to converge to the best policy. Furthermore, we introduce a method to determine the right hyperparameters for the DEQN providing system design guidance for DEQN-based DSA networks. Extensive performance evaluation confirms that DEQN-based DSA strategy is the superior choice with regard to computational power while outperforming DRQN-based DSA strategies.