Wasserstein-Barycenter Consensus for Cooperative Multi-Agent Reinforcement Learning

TL;DR

This paper introduces a novel consensus framework for cooperative multi-agent reinforcement learning using Wasserstein barycenters, promoting coherent behavior while allowing policy heterogeneity.

Contribution

It proposes a new Wasserstein barycenter-based method for policy consensus in MARL, with a derived algorithm and theoretical convergence guarantees.

Findings

Outperforms independent learners in convergence speed.

Achieves better coordination success in cooperative navigation.

Provides theoretical proof of geometric contraction of policy discrepancy.

Abstract

Cooperative multi-agent reinforcement learning (MARL) demands principled mechanisms to align heterogeneous policies while preserving the capacity for specialized behavior. We introduce a novel consensus framework that defines the team strategy as the entropic-regularized $p$-Wasserstein barycenter of agents' joint state--action visitation measures. By augmenting each agent's policy objective with a soft penalty proportional to its Sinkhorn divergence from this barycenter, the proposed approach encourages coherent group behavior without enforcing rigid parameter sharing. We derive an algorithm that alternates between Sinkhorn-barycenter computation and policy-gradient updates, and we prove that, under standard Lipschitz and compactness assumptions, the maximal pairwise policy discrepancy contracts at a geometric rate. Empirical evaluation on a cooperative navigation case study demonstrates that our OT-barycenter consensus outperforms an independent learners baseline in convergence speed and final coordination success.