Reinforcement Learning-based Joint User Scheduling and Link Configuration in Millimeter-wave Networks

TL;DR

This paper introduces reinforcement learning algorithms for joint user scheduling and link configuration in mmWave networks, aiming to minimize system delay through dynamic, online decision-making.

Contribution

It develops two reinforcement learning solutions, DRL and MAB-based, for complex joint scheduling and configuration in mmWave networks, demonstrating their effectiveness.

Findings

DRL achieves better delay performance.

MAB-based method trains faster.

Both methods effectively reduce system delay.

Abstract

In this paper, we develop algorithms for joint user scheduling and three types of mmWave link configuration: relay selection, codebook optimization, and beam tracking in millimeter wave (mmWave) networks. Our goal is to design an online controller that dynamically schedules users and configures their links to minimize the system delay. To solve this complex scheduling problem, we model it as a dynamic decision-making process and develop two reinforcement learning-based solutions. The first solution is based on deep reinforcement learning (DRL), which leverages the proximal policy optimization to train a neural network-based solution. Due to the potential high sample complexity of DRL, we also propose an empirical multi-armed bandit (MAB)-based solution, which decomposes the decision-making process into a sequential of sub-actions and exploits classic maxweight scheduling and Thompson sampling to decide those sub-actions. Our evaluation of the proposed solutions confirms their effectiveness in providing acceptable system delay. It also shows that the DRL-based solution has better delay performance while the MAB-based solution has a faster training process.