Personalized Federated Deep Reinforcement Learning for Heterogeneous Edge Content Caching Networks

TL;DR

This paper introduces a personalized federated deep reinforcement learning framework with a novel multi-head DQN algorithm to optimize content caching in heterogeneous edge networks, improving efficiency and adaptability.

Contribution

It proposes a new MH-DQN algorithm integrated into a personalized federated training framework for edge caching, addressing action space expansion and heterogeneity challenges.

Findings

MH-DQN outperforms traditional DRL algorithms in experiments.

Personalized federated training accelerates convergence and adapts to heterogeneity.

Framework improves cache utility and system performance.

Abstract

Proactive caching is essential for minimizing latency and improving Quality of Experience (QoE) in multi-server edge networks. Federated Deep Reinforcement Learning (FDRL) is a promising approach for developing cache policies tailored to dynamic content requests. However, FDRL faces challenges such as an expanding caching action space due to increased content numbers and difficulty in adapting global information to heterogeneous edge environments. In this paper, we propose a Personalized Federated Deep Reinforcement Learning framework for Caching, called PF-DRL-Ca, with the aim to maximize system utility while satisfying caching capability constraints. To manage the expanding action space, we employ a new DRL algorithm, Multi-head Deep Q-Network (MH-DQN), which reshapes the action output layers of DQN into a multi-head structure where each head generates a sub-dimensional action. We next integrate the proposed MH-DQN into a personalized federated training framework, employing a layer-wise approach for training to derive a personalized model that can adapt to heterogeneous environments while exploiting the global information to accelerate learning convergence. Our extensive experimental results demonstrate the superiority of MH-DQN over traditional DRL algorithms on a single server, as well as the advantages of the personal federated training architecture compared to other frameworks.