Function Approximation for Solving Stackelberg Equilibrium in Large Perfect Information Games

TL;DR

This paper introduces a novel neural network-based function approximation method for solving large general-sum extensive-form games by learning the Enforceable Payoff Frontier, enabling scalable computation of Stackelberg equilibria with performance guarantees.

Contribution

It proposes the first function approximation approach for Stackelberg extensive-form games, generalizing value functions to handle complex equilibria in large games.

Findings

Successfully scales to larger games than previous methods

Guarantees incentive compatibility in the learned equilibria

Achieves performance bounds based on approximation error

Abstract

Function approximation (FA) has been a critical component in solving large zero-sum games. Yet, little attention has been given towards FA in solving \textit{general-sum} extensive-form games, despite them being widely regarded as being computationally more challenging than their fully competitive or cooperative counterparts. A key challenge is that for many equilibria in general-sum games, no simple analogue to the state value function used in Markov Decision Processes and zero-sum games exists. In this paper, we propose learning the \textit{Enforceable Payoff Frontier} (EPF) -- a generalization of the state value function for general-sum games. We approximate the optimal \textit{Stackelberg extensive-form correlated equilibrium} by representing EPFs with neural networks and training them by using appropriate backup operations and loss functions. This is the first method that applies FA to the Stackelberg setting, allowing us to scale to much larger games while still enjoying performance guarantees based on FA error. Additionally, our proposed method guarantees incentive compatibility and is easy to evaluate without having to depend on self-play or approximate best-response oracles.