Conflux-PSRO: Effectively Leveraging Collective Advantages in Policy Space Response Oracles

TL;DR

Conflux-PSRO introduces a novel approach that adaptively selects and trains policies at the state level to leverage diversity effectively, improving Nash Equilibrium approximation in complex zero-sum games.

Contribution

It proposes a state-level adaptive policy selection and training method that fully exploits population diversity, enhancing performance and reducing exploitability in PSRO algorithms.

Findings

Significantly improves the utility of best responses.

Reduces exploitability compared to existing methods.

Enhances performance across various environments.

Abstract

Policy Space Response Oracle (PSRO) with policy population construction has been demonstrated as an effective method for approximating Nash Equilibrium (NE) in zero-sum games. Existing studies have attempted to improve diversity in policy space, primarily by incorporating diversity regularization into the Best Response (BR). However, these methods cause the BR to deviate from maximizing rewards, easily resulting in a population that favors diversity over performance, even when diversity is not always necessary. Consequently, exploitability is difficult to reduce until policies are fully explored, especially in complex games. In this paper, we propose Conflux-PSRO, which fully exploits the diversity of the population by adaptively selecting and training policies at state-level. Specifically, Conflux-PSRO identifies useful policies from the existing population and employs a routing policy to select the most appropriate policies at each decision point, while simultaneously training them to enhance their effectiveness. Compared to the single-policy BR of traditional PSRO and its diversity-improved variants, the BR generated by Conflux-PSRO not only leverages the specialized expertise of diverse policies but also synergistically enhances overall performance. Our experiments on various environments demonstrate that Conflux-PSRO significantly improves the utility of BRs and reduces exploitability compared to existing methods.