Pareto Deterministic Policy Gradients and Its Application in 5G Massive MIMO Networks

TL;DR

This paper introduces Pareto deterministic policy gradients (PDPG), a reinforcement learning method for optimizing cell load balance and throughput in 5G Massive MIMO networks by learning user mobility and network dynamics.

Contribution

It proposes a novel RL algorithm that handles multi-objective optimization with vector rewards, reducing the need for handcrafted scalar rewards and cross-validation, specifically applied to 5G network management.

Findings

RL method outperforms scalar-reward approaches

RL achieves performance comparable to an ideal brute-force solver

Converges reliably under different user mobility scenarios

Abstract

In this paper, we consider jointly optimizing cell load balance and network throughput via a reinforcement learning (RL) approach, where inter-cell handover (i.e., user association assignment) and massive MIMO antenna tilting are configured as the RL policy to learn. Our rationale behind using RL is to circumvent the challenges of analytically modeling user mobility and network dynamics. To accomplish this joint optimization, we integrate vector rewards into the RL value network and conduct RL action via a separate policy network. We name this method as Pareto deterministic policy gradients (PDPG). It is an actor-critic, model-free and deterministic policy algorithm which can handle the coupling objectives with the following two merits: 1) It solves the optimization via leveraging the degree of freedom of vector reward as opposed to choosing handcrafted scalar-reward; 2) Cross-validation over multiple policies can be significantly reduced. Accordingly, the RL enabled network behaves in a self-organized way: It learns out the underlying user mobility through measurement history to proactively operate handover and antenna tilt without environment assumptions. Our numerical evaluation demonstrates that the introduced RL method outperforms scalar-reward based approaches. Meanwhile, to be self-contained, an ideal static optimization based brute-force search solver is included as a benchmark. The comparison shows that the RL approach performs as well as this ideal strategy, though the former one is constrained with limited environment observations and lower action frequency, whereas the latter ones have full access to the user mobility. The convergence of our introduced approach is also tested under different user mobility environment based on our measurement data from a real scenario.