Generalized Per-Agent Advantage Estimation for Multi-Agent Policy Optimization

TL;DR

This paper introduces GPAE, a novel multi-agent reinforcement learning framework that improves sample efficiency and coordination by accurately estimating per-agent advantages using a stable, off-policy compatible method.

Contribution

The paper presents GPAE, a new advantage estimator employing a per-agent value iteration and importance sampling scheme, advancing multi-agent policy optimization.

Findings

Outperforms existing methods on benchmark tasks.

Enhances coordination in multi-agent scenarios.

Improves sample efficiency significantly.

Abstract

In this paper, we propose a novel framework for multi-agent reinforcement learning that enhances sample efficiency and coordination through accurate per-agent advantage estimation. The core of our approach is Generalized Per-Agent Advantage Estimator (GPAE), which employs a per-agent value iteration operator to compute precise per-agent advantages. This operator enables stable off-policy learning by indirectly estimating values via action probabilities, eliminating the need for direct Q-function estimation. To further refine estimation, we introduce a double-truncated importance sampling ratio scheme. This scheme improves credit assignment for off-policy trajectories by balancing sensitivity to the agent's own policy changes with robustness to non-stationarity from other agents. Experiments on benchmarks demonstrate that our approach outperforms existing approaches, excelling in coordination and sample efficiency for complex scenarios.