Asynchronous Credit Assignment for Multi-Agent Reinforcement Learning

TL;DR

This paper introduces an asynchronous credit assignment framework for multi-agent reinforcement learning that handles real-world asynchrony among agents, using a Virtual Synchrony Proxy and a multiplicative value decomposition to improve performance and interpretability.

Contribution

It proposes a novel asynchronous credit assignment framework with VSP and MVD, addressing the challenge of asynchrony in MARL and providing theoretical and empirical improvements.

Findings

Outperforms state-of-the-art MARL methods on challenging tasks.

Theoretically preserves task equilibrium and convergence.

Enhances interpretability of asynchronous cooperation.

Abstract

Credit assignment is a critical problem in multi-agent reinforcement learning (MARL), aiming to identify agents' marginal contributions for optimizing cooperative policies. Current credit assignment methods typically assume synchronous decision-making among agents. However, many real-world scenarios require agents to act asynchronously without waiting for others. This asynchrony introduces conditional dependencies between actions, which pose great challenges to current methods. To address this issue, we propose an asynchronous credit assignment framework, incorporating a Virtual Synchrony Proxy (VSP) mechanism and a Multiplicative Value Decomposition (MVD) algorithm. VSP enables physically asynchronous actions to be virtually synchronized during credit assignment. We theoretically prove that VSP preserves both task equilibrium and algorithm convergence. Furthermore, MVD leverages multiplicative interactions to effectively model dependencies among asynchronous actions, offering theoretical advantages in handling asynchronous tasks. Extensive experiments show that our framework consistently outperforms state-of-the-art MARL methods on challenging tasks while providing improved interpretability for asynchronous cooperation.