Sample-Efficient Learning of Correlated Equilibria in Extensive-Form Games

TL;DR

This paper introduces the first sample-efficient algorithm for learning Extensive-Form Correlated Equilibria (EFCE) in imperfect-information games using bandit feedback, by generalizing to $K$-EFCE and designing an uncoupled no-regret learning approach.

Contribution

It proposes a novel $K$-EFCE framework and develops the first sample-efficient bandit feedback algorithm for learning EFCE in extensive-form games.

Findings

Achieves $ ilde{O}(rac{ ext{max}_i X_i A_i^{K+1}}{ ext{epsilon}^2})$ sample complexity.

Introduces a generalized $K$-EFCE concept encompassing EFCE at $K=1$.

Provides an uncoupled no-regret algorithm for $K$-EFCE.

Abstract

Imperfect-Information Extensive-Form Games (IIEFGs) is a prevalent model for real-world games involving imperfect information and sequential plays. The Extensive-Form Correlated Equilibrium (EFCE) has been proposed as a natural solution concept for multi-player general-sum IIEFGs. However, existing algorithms for finding an EFCE require full feedback from the game, and it remains open how to efficiently learn the EFCE in the more challenging bandit feedback setting where the game can only be learned by observations from repeated playing. This paper presents the first sample-efficient algorithm for learning the EFCE from bandit feedback. We begin by proposing $K$-EFCE -- a more generalized definition that allows players to observe and deviate from the recommended actions for $K$ times. The $K$-EFCE includes the EFCE as a special case at $K=1$, and is an increasingly stricter notion of equilibrium as $K$ increases. We then design an uncoupled no-regret algorithm that finds an $\varepsilon$-approximate $K$-EFCE within $\widetilde{\mathcal{O}}(\max_{i}X_iA_i^{K}/\varepsilon^2)$ iterations in the full feedback setting, where $X_i$ and $A_i$ are the number of information sets and actions for the $i$-th player. Our algorithm works by minimizing a wide-range regret at each information set that takes into account all possible recommendation histories. Finally, we design a sample-based variant of our algorithm that learns an $\varepsilon$-approximate $K$-EFCE within $\widetilde{\mathcal{O}}(\max_{i}X_iA_i^{K+1}/\varepsilon^2)$ episodes of play in the bandit feedback setting. When specialized to $K=1$, this gives the first sample-efficient algorithm for learning EFCE from bandit feedback.