Toward Safe and Accelerated Deep Reinforcement Learning for Next-Generation Wireless Networks

TL;DR

This paper discusses the challenges and solutions for making deep reinforcement learning practical for wireless network resource management, focusing on safety and speed of convergence.

Contribution

It identifies key challenges in deploying DRL for wireless RRM and reviews approaches to enhance safety and accelerate learning, including transfer learning and reward design.

Findings

Transfer learning accelerates DRL convergence in RAN slicing.

Hybrid transfer learning improves training efficiency.

Sigmoid rewards enable safer exploration in DRL.

Abstract

Deep reinforcement learning (DRL) algorithms have recently gained wide attention in the wireless networks domain. They are considered promising approaches for solving dynamic radio resource management (RRM) problems in next-generation networks. Given their capabilities to build an approximate and continuously updated model of the wireless network environments, DRL algorithms can deal with the multifaceted complexity of such environments. Nevertheless, several challenges hinder the practical adoption of DRL in commercial networks. In this article, we first discuss two key practical challenges that are faced but rarely tackled when developing DRL-based RRM solutions. We argue that it is inevitable to address these DRL-related challenges for DRL to find its way to RRM commercial solutions. In particular, we discuss the need to have safe and accelerated DRL-based RRM solutions that mitigate the slow convergence and performance instability exhibited by DRL algorithms. We then review and categorize the main approaches used in the RRM domain to develop safe and accelerated DRL-based solutions. Finally, a case study is conducted to demonstrate the importance of having safe and accelerated DRL-based RRM solutions. We employ multiple variants of transfer learning (TL) techniques to accelerate the convergence of intelligent radio access network (RAN) slicing DRL-based controllers. We also propose a hybrid TL-based approach and sigmoid function-based rewards as examples of safe exploration in DRL-based RAN slicing.