Decentralized Federated Reinforcement Learning for User-Centric Dynamic TFDD Control

TL;DR

This paper introduces a decentralized federated reinforcement learning approach for dynamic time-frequency resource allocation in 5G networks, improving system sum rate while managing heterogeneous traffic demands and interference.

Contribution

It proposes a novel federated RL algorithm combining DDPG and Wolpertinger policy for decentralized, scalable resource management in mobile networks.

Findings

Outperforms benchmark algorithms in sum rate.

Effectively manages heterogeneous traffic demands.

Reduces inter-cell interference.

Abstract

The explosive growth of dynamic and heterogeneous data traffic brings great challenges for 5G and beyond mobile networks. To enhance the network capacity and reliability, we propose a learning-based dynamic time-frequency division duplexing (D-TFDD) scheme that adaptively allocates the uplink and downlink time-frequency resources of base stations (BSs) to meet the asymmetric and heterogeneous traffic demands while alleviating the inter-cell interference. We formulate the problem as a decentralized partially observable Markov decision process (Dec-POMDP) that maximizes the long-term expected sum rate under the users' packet dropping ratio constraints. In order to jointly optimize the global resources in a decentralized manner, we propose a federated reinforcement learning (RL) algorithm named federated Wolpertinger deep deterministic policy gradient (FWDDPG) algorithm. The BSs decide their local time-frequency configurations through RL algorithms and achieve global training via exchanging local RL models with their neighbors under a decentralized federated learning framework. Specifically, to deal with the large-scale discrete action space of each BS, we adopt a DDPG-based algorithm to generate actions in a continuous space, and then utilize Wolpertinger policy to reduce the mapping errors from continuous action space back to discrete action space. Simulation results demonstrate the superiority of our proposed algorithm to benchmark algorithms with respect to system sum rate.