To RL or not to RL? An Algorithmic Cheat-Sheet for AI-Based Radio Resource Management

TL;DR

This paper provides a comprehensive overview of various algorithmic techniques, including reinforcement learning, for radio resource management, emphasizing the importance of choosing the right method based on problem characteristics.

Contribution

It offers a structured decision framework guiding the selection of RRM algorithms, comparing RL with other techniques like optimization and MPC.

Findings

RL can be sample inefficient for RRM tasks

Different techniques suit different planning horizons and model knowledge levels

MPC presents promising future research opportunities

Abstract

Several Radio Resource Management (RRM) use cases can be framed as sequential decision planning problems, where an agent (the base station, typically) makes decisions that influence the network utility and state. While Reinforcement Learning (RL) in its general form can address this scenario, it is known to be sample inefficient. Following the principle of Occam's razor, we argue that the choice of the solution technique for RRM should be guided by questions such as, "Is it a short or long-term planning problem?", "Is the underlying model known or does it need to be learned?", "Can we solve the problem analytically?" or "Is an expert-designed policy available?". A wide range of techniques exists to address these questions, including static and stochastic optimization, bandits, model predictive control (MPC) and, indeed, RL. We review some of these techniques that have already been successfully applied to RRM, and we believe that others, such as MPC, may present exciting research opportunities for the future.