Computing Ex Ante Equilibrium in Heterogeneous Zero-Sum Team Games

TL;DR

This paper introduces H-PSRO, a novel framework for computing ex ante equilibrium in heterogeneous two-team zero-sum games, overcoming limitations of existing methods by improving policy expressiveness and convergence.

Contribution

The paper proposes H-PSRO, the first PSRO framework tailored for heterogeneous team games, with theoretical guarantees and empirical success in complex game settings.

Findings

H-PSRO achieves lower exploitability than Team PSRO.

H-PSRO converges in matrix heterogeneous games where others fail.

H-PSRO outperforms non-heterogeneous baselines in diverse settings.

Abstract

The ex ante equilibrium for two-team zero-sum games, where agents within each team collaborate to compete against the opposing team, is known to be the best a team can do for coordination. Many existing works on ex ante equilibrium solutions are aiming to extend the scope of ex ante equilibrium solving to large-scale team games based on Policy Space Response Oracle (PSRO). However, the joint team policy space constructed by the most prominent method, Team PSRO, cannot cover the entire team policy space in heterogeneous team games where teammates play distinct roles. Such insufficient policy expressiveness causes Team PSRO to be trapped into a sub-optimal ex ante equilibrium with significantly higher exploitability and never converges to the global ex ante equilibrium. To find the global ex ante equilibrium without introducing additional computational complexity, we first parameterize heterogeneous policies for teammates, and we prove that optimizing the heterogeneous teammates' policies sequentially can guarantee a monotonic improvement in team rewards. We further propose Heterogeneous-PSRO (H-PSRO), a novel framework for heterogeneous team games, which integrates the sequential correlation mechanism into the PSRO framework and serves as the first PSRO framework for heterogeneous team games. We prove that H-PSRO achieves lower exploitability than Team PSRO in heterogeneous team games. Empirically, H-PSRO achieves convergence in matrix heterogeneous games that are unsolvable by non-heterogeneous baselines. Further experiments reveal that H-PSRO outperforms non-heterogeneous baselines in both heterogeneous team games and homogeneous settings.