Multi-level Advantage Credit Assignment for Cooperative Multi-Agent Reinforcement Learning

TL;DR

This paper introduces MACA, a multi-level advantage framework for cooperative multi-agent reinforcement learning that explicitly reasons about agent contributions at various cooperation levels, improving performance on complex tasks.

Contribution

The paper proposes a novel multi-level advantage formulation with explicit counterfactual reasoning and an attention-based framework for better credit assignment in MARL.

Findings

MACA outperforms existing methods on Starcraft v1&v2 tasks.

Multi-level advantages improve credit attribution accuracy.

Explicit reasoning across cooperation levels enhances policy learning.

Abstract

Cooperative multi-agent reinforcement learning (MARL) aims to coordinate multiple agents to achieve a common goal. A key challenge in MARL is credit assignment, which involves assessing each agent's contribution to the shared reward. Given the diversity of tasks, agents may perform different types of coordination, with rewards attributed to diverse and often overlapping agent subsets. In this work, we formalize the credit assignment level as the number of agents cooperating to obtain a reward, and address scenarios with multiple coexisting levels. We introduce a multi-level advantage formulation that performs explicit counterfactual reasoning to infer credits across distinct levels. Our method, Multi-level Advantage Credit Assignment (MACA), captures agent contributions at multiple levels by integrating advantage functions that reason about individual, joint, and correlated actions. Utilizing an attention-based framework, MACA identifies correlated agent relationships and constructs multi-level advantages to guide policy learning. Comprehensive experiments on challenging Starcraft v1\&v2 tasks demonstrate MACA's superior performance, underscoring its efficacy in complex credit assignment scenarios.