On Centralized Critics in Multi-Agent Reinforcement Learning

TL;DR

This paper provides a theoretical and empirical analysis of centralized critics in multi-agent reinforcement learning, revealing potential drawbacks of state-based critics and their impact on learning under partial observability.

Contribution

It offers a formal analysis showing that centralized critics are not always beneficial and that state-based critics can introduce bias and variance, challenging common assumptions.

Findings

Centralized critics are not always advantageous in MARL.

State-based critics can cause bias and variance issues.

Practical experiments highlight challenges with partial observability.

Abstract

Centralized Training for Decentralized Execution where agents are trained offline in a centralized fashion and execute online in a decentralized manner, has become a popular approach in Multi-Agent Reinforcement Learning (MARL). In particular, it has become popular to develop actor-critic methods that train decentralized actors with a centralized critic where the centralized critic is allowed access global information of the entire system, including the true system state. Such centralized critics are possible given offline information and are not used for online execution. While these methods perform well in a number of domains and have become a de facto standard in MARL, using a centralized critic in this context has yet to be sufficiently analyzed theoretically or empirically. In this paper, we therefore formally analyze centralized and decentralized critic approaches, and analyze the effect of using state-based critics in partially observable environments. We derive theories contrary to the common intuition: critic centralization is not strictly beneficial, and using state values can be harmful. We further prove that, in particular, state-based critics can introduce unexpected bias and variance compared to history-based critics. Finally, we demonstrate how the theory applies in practice by comparing different forms of critics on a wide range of common multi-agent benchmarks. The experiments show practical issues such as the difficulty of representation learning with partial observability, which highlights why the theoretical problems are often overlooked in the literature.