Cooperative Game-Theoretic Credit Assignment for Multi-Agent Policy Gradients via the Core

TL;DR

This paper introduces CORA, a novel multi-agent reinforcement learning method that allocates credit based on cooperative game theory, improving coordination and performance by considering coalition contributions.

Contribution

We propose a coalition-based advantage allocation method using the core from cooperative game theory, enhancing credit assignment in multi-agent policy optimization.

Findings

CORA outperforms baseline methods on various multi-agent benchmarks.

The core-based allocation improves coordination among agents.

Random coalition sampling reduces computational costs effectively.

Abstract

This work focuses on the credit assignment problem in cooperative multi-agent reinforcement learning (MARL). Sharing the global advantage among agents often leads to insufficient policy optimization, as it fails to capture the coalitional contributions of different agents. In this work, we revisit the policy update process from a coalitional perspective and propose CORA, an advantage allocation method guided by a cooperative game-theoretic core allocation. By evaluating the marginal contributions of different coalitions and combining clipped double Q-learning to mitigate overestimation bias, CORA estimates coalition-wise advantages. The core formulation enforces coalition-wise lower bounds on allocated credits, so that coalitions with higher advantages receive stronger total incentives for their participating agents, enabling the global advantage to be attributed to different coalition strategies and promoting coordinated optimal behavior. To reduce computational overhead, we employ random coalition sampling to approximate the core allocation efficiently. Experiments on matrix games, differential games, and multi-agent collaboration benchmarks demonstrate that our method outperforms baselines. These findings highlight the importance of coalition-level credit assignment and cooperative games for advancing multi-agent learning.