Taming Multi-Agent Reinforcement Learning with Estimator Variance Reduction

TL;DR

This paper introduces PERLA, a variance reduction technique for multi-agent reinforcement learning that improves the accuracy of value estimates, enhances scalability, and maintains the benefits of centralized training.

Contribution

PERLA is a novel framework that reduces estimator variance in CT-DE MARL methods by sampling joint-policy estimates, enabling more efficient and scalable learning.

Findings

PERLA significantly reduces variance in value estimates.

PERLA improves learning efficiency in multi-agent benchmarks.

PERLA outperforms baseline methods in Multi-agent Mujoco and StarCraft II.

Abstract

Centralised training with decentralised execution (CT-DE) serves as the foundation of many leading multi-agent reinforcement learning (MARL) algorithms. Despite its popularity, it suffers from a critical drawback due to its reliance on learning from a single sample of the joint-action at a given state. As agents explore and update their policies during training, these single samples may poorly represent the actual joint-policy of the system of agents leading to high variance gradient estimates that hinder learning. To address this problem, we propose an enhancement tool that accommodates any actor-critic MARL method. Our framework, Performance Enhancing Reinforcement Learning Apparatus (PERLA), introduces a sampling technique of the agents' joint-policy into the critics while the agents train. This leads to TD updates that closely approximate the true expected value under the current joint-policy rather than estimates from a single sample of the joint-action at a given state. This produces low variance and precise estimates of expected returns, minimising the variance in the critic estimators which typically hinders learning. Moreover, as we demonstrate, by eliminating much of the critic variance from the single sampling of the joint policy, PERLA enables CT-DE methods to scale more efficiently with the number of agents. Theoretically, we prove that PERLA reduces variance in value estimates similar to that of decentralised training while maintaining the benefits of centralised training. Empirically, we demonstrate PERLA's superior performance and ability to reduce estimator variance in a range of benchmarks including Multi-agent Mujoco, and StarCraft II Multi-agent Challenge.