When Deep Reinforcement Learning Meets Federated Learning: Intelligent Multi-Timescale Resource Management for Multi-access Edge Computing in 5G Ultra Dense Network

TL;DR

This paper introduces an intelligent framework for ultra-dense edge computing in 5G networks, combining blockchain, AI, and federated learning with a novel two-timescale deep reinforcement learning approach to optimize resource management and protect privacy.

Contribution

It proposes a new I-UDEC framework integrating blockchain, AI, and federated learning, along with a novel 2Ts-DRL method for efficient, privacy-preserving resource management in 5G edge networks.

Findings

Reduces task execution time by up to 31.87%.

Demonstrates effectiveness of 2Ts-DRL and federated learning in resource optimization.

Enhances privacy protection in edge computing through federated learning.

Abstract

Ultra-dense edge computing (UDEC) has great potential, especially in the 5G era, but it still faces challenges in its current solutions, such as the lack of: i) efficient utilization of multiple 5G resources (e.g., computation, communication, storage and service resources); ii) low overhead offloading decision making and resource allocation strategies; and iii) privacy and security protection schemes. Thus, we first propose an intelligent ultra-dense edge computing (I-UDEC) framework, which integrates blockchain and Artificial Intelligence (AI) into 5G ultra-dense edge computing networks. First, we show the architecture of the framework. Then, in order to achieve real-time and low overhead computation offloading decisions and resource allocation strategies, we design a novel two-timescale deep reinforcement learning (\textit{2Ts-DRL}) approach, consisting of a fast-timescale and a slow-timescale learning process, respectively. The primary objective is to minimize the total offloading delay and network resource usage by jointly optimizing computation offloading, resource allocation and service caching placement. We also leverage federated learning (FL) to train the \textit{2Ts-DRL} model in a distributed manner, aiming to protect the edge devices' data privacy. Simulation results corroborate the effectiveness of both the \textit{2Ts-DRL} and FL in the I-UDEC framework and prove that our proposed algorithm can reduce task execution time up to 31.87%.