Redistributing Rewards Across Time and Agents for Multi-Agent Reinforcement Learning

TL;DR

This paper introduces TAR$^2$, a novel method for credit assignment in multi-agent reinforcement learning that decouples reward modeling from normalization, ensuring policy preservation and improving learning efficiency.

Contribution

We propose TAR$^2$, which learns contribution scores separately from normalization, guaranteeing return equivalence and policy preservation without relying on model accuracy.

Findings

TAR$^2$ accelerates learning on SMACLite and GRF benchmarks.

TAR$^2$ achieves higher final performance than strong baselines.

The method guarantees policy preservation regardless of model accuracy.

Abstract

Credit assignmen, disentangling each agent's contribution to a shared reward, is a critical challenge in cooperative multi-agent reinforcement learning (MARL). To be effective, credit assignment methods must preserve the environment's optimal policy. Some recent approaches attempt this by enforcing return equivalence, where the sum of distributed rewards must equal the team reward. However, their guarantees are conditional on a learned model's regression accuracy, making them unreliable in practice. We introduce Temporal-Agent Reward Redistribution (TAR$^2$), an approach that decouples credit modeling from this constraint. A neural network learns unnormalized contribution scores, while a separate, deterministic normalization step enforces return equivalence by construction. We demonstrate that this method is equivalent to a valid Potential-Based Reward Shaping (PBRS), which guarantees the optimal policy is preserved regardless of model accuracy. Empirically, on challenging SMACLite and Google Research Football (GRF) benchmarks, TAR$^2$ accelerates learning and achieves higher final performance than strong baselines. These results establish our method as an effective solution for the agent-temporal credit assignment problem.