On Stateful Value Factorization in Multi-Agent Reinforcement Learning

TL;DR

This paper analyzes the theoretical foundations of state versus history in value factorization for multi-agent reinforcement learning, introduces DuelMIX for improved utility estimation, and demonstrates its effectiveness in complex tasks.

Contribution

It provides a formal analysis reconciling theory and practice in state-based value factorization and proposes DuelMIX, a novel algorithm with enhanced expressiveness.

Findings

DuelMIX outperforms existing methods on StarCraft II tasks.

Using state information improves the theoretical and practical performance.

The approach achieves full expressiveness in utility estimation.

Abstract

Value factorization is a popular paradigm for designing scalable multi-agent reinforcement learning algorithms. However, current factorization methods make choices without full justification that may limit their performance. For example, the theory in prior work uses stateless (i.e., history) functions, while the practical implementations use state information -- making the motivating theory a mismatch for the implementation. Also, methods have built off of previous approaches, inheriting their architectures without exploring other, potentially better ones. To address these concerns, we formally analyze the theory of using the state instead of the history in current methods -- reconnecting theory and practice. We then introduce DuelMIX, a factorization algorithm that learns distinct per-agent utility estimators to improve performance and achieve full expressiveness. Experiments on StarCraft II micromanagement and Box Pushing tasks demonstrate the benefits of our intuitions.