Deep Reinforcement Learning for Dynamic Multichannel Access in Wireless Networks

TL;DR

This paper introduces a deep reinforcement learning approach using Deep Q-Networks to optimize multichannel access in wireless networks with unknown dynamics, achieving near-optimal performance without prior system knowledge.

Contribution

It presents a novel application of DQN to dynamic multichannel access, including an adaptive method for time-varying environments, outperforming traditional heuristics.

Findings

DQN achieves near-optimal performance in complex scenarios.

The adaptive DQN effectively handles time-varying dynamics.

DQN outperforms Myopic and Whittle Index policies in simulations.

Abstract

We consider a dynamic multichannel access problem, where multiple correlated channels follow an unknown joint Markov model. A user at each time slot selects a channel to transmit data and receives a reward based on the success or failure of the transmission. The objective is to find a policy that maximizes the expected long-term reward. The problem is formulated as a partially observable Markov decision process (POMDP) with unknown system dynamics. To overcome the challenges of unknown system dynamics as well as prohibitive computation, we apply the concept of reinforcement learning and implement a Deep Q-Network (DQN) that can deal with large state space without any prior knowledge of the system dynamics. We provide an analytical study on the optimal policy for fixed-pattern channel switching with known system dynamics and show through simulations that DQN can achieve the same optimal performance without knowing the system statistics. We compare the performance of DQN with a Myopic policy and a Whittle Index-based heuristic through both simulations as well as real-data trace and show that DQN achieves near-optimal performance in more complex situations. Finally, we propose an adaptive DQN approach with the capability to adapt its learning in time-varying, dynamic scenarios.