Embedded Mean Field Reinforcement Learning for Perimeter-defense Game

TL;DR

This paper develops a scalable reinforcement learning framework, EMFAC, for large-scale, realistic perimeter-defense scenarios involving heterogeneous UAVs, motion dynamics, and environmental factors, validated through extensive simulations and real-world tests.

Contribution

It introduces EMFAC, a novel mean-field reinforcement learning approach with attention mechanisms, tailored for complex, large-scale perimeter-defense tasks with heterogeneous agents and realistic dynamics.

Findings

EMFAC outperforms baseline methods in convergence speed and performance.

The framework effectively handles large-scale heterogeneous control challenges.

Real-world experiments confirm practical applicability and robustness.

Abstract

With the rapid advancement of unmanned aerial vehicles (UAVs) and missile technologies, perimeter-defense game between attackers and defenders for the protection of critical regions have become increasingly complex and strategically significant across a wide range of domains. However, existing studies predominantly focus on small-scale, simplified two-dimensional scenarios, often overlooking realistic environmental perturbations, motion dynamics, and inherent heterogeneity--factors that pose substantial challenges to real-world applicability. To bridge this gap, we investigate large-scale heterogeneous perimeter-defense game in a three-dimensional setting, incorporating realistic elements such as motion dynamics and wind fields. We derive the Nash equilibrium strategies for both attackers and defenders, characterize the victory regions, and validate our theoretical findings through extensive simulations. To tackle large-scale heterogeneous control challenges in defense strategies, we propose an Embedded Mean-Field Actor-Critic (EMFAC) framework. EMFAC leverages representation learning to enable high-level action aggregation in a mean-field manner, supporting scalable coordination among defenders. Furthermore, we introduce a lightweight agent-level attention mechanism based on reward representation, which selectively filters observations and mean-field information to enhance decision-making efficiency and accelerate convergence in large-scale tasks. Extensive simulations across varying scales demonstrate the effectiveness and adaptability of EMFAC, which outperforms established baselines in both convergence speed and overall performance. To further validate practicality, we test EMFAC in small-scale real-world experiments and conduct detailed analyses, offering deeper insights into the framework's effectiveness in complex scenarios.