Asynchronous Fractional Multi-Agent Deep Reinforcement Learning for Age-Minimal Mobile Edge Computing

TL;DR

This paper introduces a novel asynchronous fractional multi-agent deep reinforcement learning framework to optimize task scheduling and offloading in mobile edge computing, significantly reducing the Age of Information in real-time networked applications.

Contribution

It develops a fractional RL approach for multi-agent systems, extending Dinkelbach's method, and applies it to minimize AoI in MEC with asynchronous decision-making, achieving linear convergence to Nash equilibrium.

Findings

Reduces average AoI by up to 50.6% compared to baselines.

Establishes linear convergence rate for the proposed fractional RL framework.

Provides conditions for convergence to Nash equilibrium in asynchronous settings.

Abstract

In the realm of emerging real-time networked applications such as cyber-physical systems (CPS), the Age of Information (AoI) has emerged as a pivotal metric for evaluating timeliness. To meet the high computational demands, such as those in smart manufacturing within CPS, mobile edge computing (MEC) presents a promising solution for optimizing computing and reducing AoI. In this work, we study the timeliness of compute-intensive updates and explore jointly optimizing the task updating (when to generate a task) and offloading (where to process a task) policies to minimize AoI. Specifically, we consider edge load dynamics and formulate a task scheduling problem to minimize the expected time-average AoI. Solving this problem is challenging due to the fractional objective introduced by AoI and the asynchronous decision-making of the semi-Markov game (SMG). To this end, we propose a fractional reinforcement learning (RL) framework. We begin by introducing a fractional single-agent RL framework and establish its linear convergence rate. Building on this, we develop a fractional multi-agent RL framework, extend Dinkelbach's method, and demonstrate its equivalence to the inexact Newton's method. Furthermore, we provide the conditions under which the framework achieves linear convergence to the Nash equilibrium (NE). To tackle the challenge of asynchronous decision-making in the SMG, we further design an asynchronous model-free fractional multi-agent RL algorithm, where each mobile device can determine the task updating and offloading decisions without knowing the real-time system dynamics and decisions of other devices. Experimental results show that when compared with the best existing baseline algorithm, our proposed algorithm reduces the average AoI by up to 50.6%.