Addressing Rotational Learning Dynamics in Multi-Agent Reinforcement Learning

TL;DR

This paper identifies rotational optimization dynamics as a key challenge in multi-agent reinforcement learning and proposes a variational inequality framework with gradient-based methods to improve convergence and coordination.

Contribution

It introduces a unified VI-based framework for MARL, integrating advanced optimization techniques to address rotational dynamics and improve algorithm performance.

Findings

Enhanced convergence to equilibrium in zero-sum games

Improved team coordination in multi-agent environments

Significant performance gains across benchmarks

Abstract

Multi-agent reinforcement learning (MARL) has emerged as a powerful paradigm for solving complex problems through agents' cooperation and competition, finding widespread applications across domains. Despite its success, MARL faces a reproducibility crisis. We show that, in part, this issue is related to the rotational optimization dynamics arising from competing agents' objectives, and require methods beyond standard optimization algorithms. We reframe MARL approaches using Variational Inequalities (VIs), offering a unified framework to address such issues. Leveraging optimization techniques designed for VIs, we propose a general approach for integrating gradient-based VI methods capable of handling rotational dynamics into existing MARL algorithms. Empirical results demonstrate significant performance improvements across benchmarks. In zero-sum games, Rock--paper--scissors and Matching pennies, VI methods achieve better convergence to equilibrium strategies, and in the Multi-Agent Particle Environment: Predator-prey, they also enhance team coordination. These results underscore the transformative potential of advanced optimization techniques in MARL.