A-PSRO: A Unified Strategy Learning Method with Advantage Function for Normal-form Games

TL;DR

A-PSRO is a unified open-ended learning framework for normal-form games that uses an advantage function to improve strategy learning, reducing exploitability and increasing rewards in different game types.

Contribution

It introduces the advantage function into the PSRO framework, providing a unified learning objective for both zero-sum and general-sum games, with theoretical and empirical validation.

Findings

Reduces exploitability in zero-sum games

Increases rewards in general-sum games

Outperforms previous PSRO algorithms

Abstract

Solving Nash equilibrium is the key challenge in normal-form games with large strategy spaces, where open-ended learning frameworks offer an efficient approach. In this work, we propose an innovative unified open-ended learning framework A-PSRO, i.e., Advantage Policy Space Response Oracle, as a comprehensive framework for both zero-sum and general-sum games. In particular, we introduce the advantage function as an enhanced evaluation metric for strategies, enabling a unified learning objective for agents engaged in normal-form games. We prove that the advantage function exhibits favorable properties and is connected with the Nash equilibrium, which can be used as an objective to guide agents to learn strategies efficiently. Our experiments reveal that A-PSRO achieves a considerable decrease in exploitability in zero-sum games and an escalation in rewards in general-sum games, significantly outperforming previous PSRO algorithms.