MAPPO-LCR: Multi-Agent Proximal Policy Optimization with Local Cooperation Reward in Spatial Public Goods Games

TL;DR

This paper introduces MAPPO-LCR, a novel multi-agent reinforcement learning framework that enhances cooperation in spatial public goods games by incorporating local cooperation rewards and centralized critics.

Contribution

It pioneers the application of MAPPO with local cooperation rewards to spatial public goods games, addressing payoff coupling and non-stationarity issues in large populations.

Findings

MAPPO-LCR achieves stable cooperation emergence.

It converges reliably across various enhancement factors.

It outperforms PPO in learning efficiency and cooperation levels.

Abstract

Spatial public goods games model collective dilemmas where individual payoffs depend on population-level strategy configurations. Most existing studies rely on evolutionary update rules or value-based reinforcement learning methods. These approaches struggle to represent payoff coupling and non-stationarity in large interacting populations. This work introduces Multi-Agent Proximal Policy Optimization (MAPPO) into spatial public goods games for the first time. In these games, individual returns are intrinsically coupled through overlapping group interactions. Proximal Policy Optimization (PPO) treats agents as independent learners and ignores this coupling during value estimation. MAPPO addresses this limitation through a centralized critic that evaluates joint strategy configurations. To study neighborhood-level cooperation signals under this framework, we propose MAPPO with Local Cooperation Reward, termed MAPPO-LCR. The local cooperation reward aligns policy updates with surrounding cooperative density without altering the original game structure. MAPPO-LCR preserves decentralized execution while enabling population-level value estimation during training. Extensive simulations demonstrate stable cooperation emergence and reliable convergence across enhancement factors. Statistical analyses further confirm the learning advantage of MAPPO over PPO in spatial public goods games.