Value-Set Iteration: Computing Optimal Correlated Equilibria in Infinite-Horizon Multi-Player Stochastic Games

TL;DR

This paper introduces a polynomial-time algorithm for computing near-optimal correlated equilibria in infinite-horizon multi-player stochastic games, addressing the challenge of history-dependent policies through a novel value-set iteration method.

Contribution

It presents a new polynomial-time algorithm for approximate correlated equilibria in complex stochastic games using inducible value sets and fixed point characterization.

Findings

Algorithm computes $(psilon,elta)$-optimal CEs efficiently

Complexity is polynomial in key parameters for general games

Turn-based games have reduced polynomial complexity

Abstract

We study the problem of computing optimal correlated equilibria (CEs) in infinite-horizon multi-player stochastic games, where correlation signals are provided over time. In this setting, optimal CEs require history-dependent policies; this poses new representational and algorithmic challenges as the number of possible histories grows exponentially with the number of time steps. We focus on computing $(\epsilon, \delta)$-optimal CEs -- solutions that achieve a value within $\epsilon$ of an optimal CE, while allowing the agents' incentive constraints to be violated by at most $\delta$. Our main result is an algorithm that computes an $(\epsilon,\delta)$-optimal CE in time polynomial in $1/(\epsilon\delta(1 - \gamma))^{n+1}$, where $\gamma$ is the discount factor, and $n$ is the number of agents. For (a slightly more general variant of) turn-based games, we further reduce the complexity to a polynomial in $n$. We also establish that the bi-criterion approximation is necessary by proving matching inapproximability bounds. Our technical core is a novel approach based on inducible value sets, which leverages a compact representation of history-dependent CEs through the values they induce to overcome the representational challenge. We develop the value-set iteration algorithm -- which operates by iteratively updating estimates of inducible value sets -- and characterize CEs as the greatest fixed point of the update map. Our algorithm provides a groundwork for computing optimal CEs in general multi-player stochastic settings.