Conservative Equilibrium Discovery in Offline Game-Theoretic Multiagent Reinforcement Learning

TL;DR

This paper introduces COffeE-PSRO, a conservative offline multiagent reinforcement learning method that improves equilibrium discovery by accounting for data uncertainty and guiding strategy exploration.

Contribution

It extends PSRO with uncertainty quantification and a novel meta-strategy solver, enabling more reliable equilibrium approximation in offline multiagent settings.

Findings

COffeE-PSRO outperforms state-of-the-art offline methods in equilibrium quality.

The approach effectively quantifies game dynamics uncertainty.

Empirical results show improved strategy stability and lower regret.

Abstract

Offline learning of strategies takes data efficiency to its extreme by restricting algorithms to a fixed dataset of state-action trajectories. We consider the problem in a mixed-motive multiagent setting, where the goal is to solve a game under the offline learning constraint. We first frame this problem in terms of selecting among candidate equilibria. Since datasets may inform only a small fraction of game dynamics, it is generally infeasible in offline game-solving to even verify a proposed solution is a true equilibrium. Therefore, we consider the relative probability of low regret (i.e., closeness to equilibrium) across candidates based on the information available. Specifically, we extend Policy Space Response Oracles (PSRO), an online game-solving approach, by quantifying game dynamics uncertainty and modifying the RL objective to skew towards solutions more likely to have low regret in the true game. We further propose a novel meta-strategy solver, tailored for the offline setting, to guide strategy exploration in PSRO. Our incorporation of Conservatism principles from Offline reinforcement learning approaches for strategy Exploration gives our approach its name: COffeE-PSRO. Experiments demonstrate COffeE-PSRO's ability to extract lower-regret solutions than state-of-the-art offline approaches and reveal relationships between algorithmic components empirical game fidelity, and overall performance.