Multi-Agent Q-Learning for Real-Time Load Balancing User Association and Handover in Mobile Networks

TL;DR

This paper introduces multi-agent online Q-learning algorithms for real-time load balancing, user association, and handover management in dense cellular networks, improving stability and performance.

Contribution

It proposes novel centralized and distributed multi-agent Q-learning policies that adapt to network dynamics and ensure load balancing during user association and handover.

Findings

Outperforms 3GPP max-SINR association in simulations.

Demonstrates robustness across various user mobility profiles.

Achieves low-complexity and fast convergence in dynamic environments.

Abstract

As next generation cellular networks become denser, associating users with the optimal base stations at each time while ensuring no base station is overloaded becomes critical for achieving stable and high network performance. We propose multi-agent online Q-learning (QL) algorithms for performing real-time load balancing user association and handover in dense cellular networks. The load balancing constraints at all base stations couple the actions of user agents, and we propose two multi-agent action selection policies, one centralized and one distributed, to satisfy load balancing at every learning step. In the centralized policy, the actions of UEs are determined by a central load balancer (CLB) running an algorithm based on swapping the worst connection to maximize the total learning reward. In the distributed policy, each UE takes an action based on its local information by participating in a distributed matching game with the BSs to maximize the local reward. We then integrate these action selection policies into an online QL algorithm that adapts in real-time to network dynamics including channel variations and user mobility, using a reward function that considers a handover cost to reduce handover frequency. The proposed multi-agent QL algorithm features low-complexity and fast convergence, outperforming 3GPP max-SINR association. Both policies adapt well to network dynamics at various UE speed profiles from walking, running, to biking and suburban driving, illustrating their robustness and real-time adaptability.