LLM-Driven Stationarity-Aware Expert Demonstrations for Multi-Agent Reinforcement Learning in Mobile Systems

TL;DR

This paper introduces RELED, a scalable multi-agent reinforcement learning framework that uses large language model-generated expert demonstrations and stationarity-aware techniques to improve training stability and policy convergence in resource-constrained mobile systems.

Contribution

The paper presents a novel MARL framework integrating LLM-driven expert demonstrations with stationarity-aware bounds for enhanced stability and scalability.

Findings

RELED outperforms existing MARL methods in real city network experiments.

The stationarity-aware module improves training stability and convergence.

Hybrid policy optimization accelerates learning and enhances generalization.

Abstract

Multi-agent reinforcement learning (MARL) has been increasingly adopted in many real-world applications. While MARL enables decentralized deployment on resource-constrained edge devices, it suffers from severe non-stationarity due to the synchronous updates of agent policies. This non stationarity results in unstable training and poor policy con vergence, especially as the number of agents increases. In this paper, we propose RELED, a scalable MARL framework that integrates large language model (LLM)-driven expert demonstrations with autonomous agent exploration. RELED incorporates a Stationarity-Aware Expert Demonstration module, which leverages theoretical non-stationarity bounds to enhance the quality of LLM-generated expert trajectories, thus providing high reward and training-stable samples for each agent. Moreover, a Hybrid Expert-Agent Policy Optimization module adaptively balances each agent's learning from both expert-generated and agent-generated trajectories, accelerating policy convergence and improving generalization. Extensive experiments with real city networks based on OpenStreetMap demonstrate that RELED achieves superior performance compared to state-of-the-art MARL methods.