LMFPPO-UBP: Local Mean Field Proximal Policy Optimization with Unbalanced Punishment for Spatial Public Goods Games

TL;DR

This paper introduces LMFPPO-UBP, a novel reinforcement learning framework that enhances cooperation in spatial public goods games by incorporating local mean-field dynamics and unbalanced punishment mechanisms.

Contribution

It reformulates the mean field as a socio-statistical sensor within policy gradients and integrates unbalanced punishment to effectively promote cooperation.

Findings

Outperforms baseline methods like Q-learning and Fermi update rules.

Promotes rapid and stable cooperation under low enhancement factors.

Reduces the cooperation threshold and improves coordination.

Abstract

Spatial public goods games are characterized by high-dimensional state spaces and localized externalities, which pose significant challenges for achieving stable and widespread cooperation. Traditional approaches often struggle to effectively capture neighborhood-level strategic interactions and dynamically align individual incentives with collective welfare. To resolve this issue, this paper introduces a novel intelligent decision-making framework called Local Mean-Field Proximal Policy Optimization with Unbalanced Punishment (LMFPPO-UBP). The conventional mean field concept is reformulated as a socio-statistical sensor embedded directly into the policy gradient space of deep reinforcement learning, allowing agents to adapt their strategies based on mesoscale neighborhood dynamics. Additionally, an unbalanced punishment mechanism is integrated to penalize defectors proportionally to the local density of cooperators, thereby reshaping the payoff structures without imposing direct costs on cooperative agents. Experimental results demonstrate that the LMFPPO-UBP promotes rapid and stable global cooperation even under low enhancement factors, consistently outperforming baseline methods such as Q-learning and Fermi update rules. Statistical analyses further validate the framework's effectiveness in lowering the cooperation threshold and achieving better coordinated outcomes.