Breakthrough the Suboptimal Stable Point in Value-Factorization-Based Multi-Agent Reinforcement Learning

TL;DR

This paper introduces the concept of stable points in value factorization for multi-agent reinforcement learning, analyzes their impact on suboptimal convergence, and proposes MRVF to iteratively eliminate suboptimal actions, improving performance.

Contribution

It presents a new theoretical framework for understanding stable points in MARL and introduces MRVF, a practical method to avoid suboptimal stable points and enhance learning outcomes.

Findings

MRVF outperforms state-of-the-art methods on benchmarks.

Analysis shows suboptimal stable points cause poor performance.

Iterative filtering of suboptimal actions improves convergence.

Abstract

Value factorization, a popular paradigm in MARL, faces significant theoretical and algorithmic bottlenecks: its tendency to converge to suboptimal solutions remains poorly understood and unsolved. Theoretically, existing analyses fail to explain this due to their primary focus on the optimal case. To bridge this gap, we introduce a novel theoretical concept: the stable point, which characterizes the potential convergence of value factorization in general cases. Through an analysis of stable point distributions in existing methods, we reveal that non-optimal stable points are the primary cause of poor performance. However, algorithmically, making the optimal action the unique stable point is nearly infeasible. In contrast, iteratively filtering suboptimal actions by rendering them unstable emerges as a more practical approach for global optimality. Inspired by this, we propose a novel Multi-Round Value Factorization (MRVF) framework. Specifically, by measuring a non-negative payoff increment relative to the previously selected action, MRVF transforms inferior actions into unstable points, thereby driving each iteration toward a stable point with a superior action. Experiments on challenging benchmarks, including predator-prey tasks and StarCraft II Multi-Agent Challenge (SMAC), validate our analysis of stable points and demonstrate the superiority of MRVF over state-of-the-art methods.