Sample-Efficient Policy Space Response Oracles with Joint Experience Best Response

TL;DR

This paper introduces Joint Experience Best Response (JBR), a method that enhances sample efficiency in multi-agent reinforcement learning by reusing data for simultaneous best response computation, making PSRO more practical for large-scale environments.

Contribution

The paper proposes JBR, a novel modification to PSRO that reuses joint trajectories for all agents' best responses, reducing environment interactions and improving efficiency.

Findings

JBR improves sample efficiency in multi-agent environments.

Exploration-Augmented JBR offers the best accuracy-efficiency trade-off.

Hybrid BR achieves near-PSRO performance with less data.

Abstract

Multi-agent reinforcement learning (MARL) offers a scalable alternative to exact game-theoretic analysis but suffers from non-stationarity and the need to maintain diverse populations of strategies that capture non-transitive interactions. Policy Space Response Oracles (PSRO) address these issues by iteratively expanding a restricted game with approximate best responses (BRs), yet per-agent BR training makes it prohibitively expensive in many-agent or simulator-expensive settings. We introduce Joint Experience Best Response (JBR), a drop-in modification to PSRO that collects trajectories once under the current meta-strategy profile and reuses this joint dataset to compute BRs for all agents simultaneously. This amortizes environment interaction and improves the sample efficiency of best-response computation. Because JBR converts BR computation into an offline RL problem, we propose three remedies for distribution-shift bias: (i) Conservative JBR with safe policy improvement, (ii) Exploration-Augmented JBR that perturbs data collection and admits theoretical guarantees, and (iii) Hybrid BR that interleaves JBR with periodic independent BR updates. Across benchmark multi-agent environments, Exploration-Augmented JBR achieves the best accuracy-efficiency trade-off, while Hybrid BR attains near-PSRO performance at a fraction of the sample cost. Overall, JBR makes PSRO substantially more practical for large-scale strategic learning while preserving equilibrium robustness.