Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization

TL;DR

This paper introduces InSPO, a novel offline multi-agent reinforcement learning algorithm that sequentially updates agent policies to improve coordination, avoid out-of-distribution actions, and ensure convergence to optimal solutions.

Contribution

The paper proposes InSPO, a new offline MARL method that sequentially updates policies in-sample, addressing coordination and OOD issues, with theoretical guarantees of monotonic improvement and convergence.

Findings

InSPO outperforms existing offline MARL methods in experiments.

InSPO guarantees monotonic policy improvement and convergence to QRE.

The method effectively addresses OOD actions and premature convergence.

Abstract

Offline Multi-Agent Reinforcement Learning (MARL) is an emerging field that aims to learn optimal multi-agent policies from pre-collected datasets. Compared to single-agent case, multi-agent setting involves a large joint state-action space and coupled behaviors of multiple agents, which bring extra complexity to offline policy optimization. In this work, we revisit the existing offline MARL methods and show that in certain scenarios they can be problematic, leading to uncoordinated behaviors and out-of-distribution (OOD) joint actions. To address these issues, we propose a new offline MARL algorithm, named In-Sample Sequential Policy Optimization (InSPO). InSPO sequentially updates each agent's policy in an in-sample manner, which not only avoids selecting OOD joint actions but also carefully considers teammates' updated policies to enhance coordination. Additionally, by thoroughly exploring low-probability actions in the behavior policy, InSPO can well address the issue of premature convergence to sub-optimal solutions. Theoretically, we prove InSPO guarantees monotonic policy improvement and converges to quantal response equilibrium (QRE). Experimental results demonstrate the effectiveness of our method compared to current state-of-the-art offline MARL methods.