Hierarchical Deep Q-Learning Based Handover in Wireless Networks with Dual Connectivity

TL;DR

This paper introduces hierarchical deep Q-learning algorithms to enhance dual connectivity handover in 5G networks, significantly reducing latency and improving performance in urban scenarios with high-frequency antennas.

Contribution

It proposes novel hierarchical deep Q-learning methods for dual connectivity handover, demonstrating superior performance over fixed and dynamic parameter baselines in 5G scenarios.

Findings

Significant latency reductions with up to 47.6% improvement.

Hierarchical Deep Q-Learning achieves more optimal solutions than single-agent methods.

Context-awareness, such as geo-location, can further enhance handover performance.

Abstract

5G New Radio proposes the usage of frequencies above 10 GHz to speed up LTE's existent maximum data rates. However, the effective size of 5G antennas and consequently its repercussions in the signal degradation in urban scenarios makes it a challenge to maintain stable coverage and connectivity. In order to obtain the best from both technologies, recent dual connectivity solutions have proved their capabilities to improve performance when compared with coexistent standalone 5G and 4G technologies. Reinforcement learning (RL) has shown its huge potential in wireless scenarios where parameter learning is required given the dynamic nature of such context. In this paper, we propose two reinforcement learning algorithms: a single agent RL algorithm named Clipped Double Q-Learning (CDQL) and a hierarchical Deep Q-Learning (HiDQL) to improve Multiple Radio Access Technology (multi-RAT) dual-connectivity handover. We compare our proposal with two baselines: a fixed parameter and a dynamic parameter solution. Simulation results reveal significant improvements in terms of latency with a gain of 47.6% and 26.1% for Digital-Analog beamforming (BF), 17.1% and 21.6% for Hybrid-Analog BF, and 24.7% and 39% for Analog-Analog BF when comparing the RL-schemes HiDQL and CDQL with the with the existent solutions, HiDQL presented a slower convergence time, however obtained a more optimal solution than CDQL. Additionally, we foresee the advantages of utilizing context-information as geo-location of the UEs to reduce the beam exploration sector, and thus improving further multi-RAT handover latency results.