Finite-horizon Approximations and Episodic Equilibrium for Stochastic Games

TL;DR

This paper introduces a finite-horizon approximation and episodic equilibrium concept for stochastic games, providing error bounds and a unifying framework for analyzing both discounted and time-averaged utilities, with decentralized learning algorithms demonstrating convergence.

Contribution

It proposes a novel finite-horizon approximation scheme and episodic equilibrium for stochastic games, bridging finite and infinite-horizon analysis, and develops decentralized learning dynamics that converge to these equilibria.

Findings

Error bounds decay with episode length

Decentralized learning reaches episodic equilibrium

Applicable to various classes of stochastic games

Abstract

This paper proposes a finite-horizon approximation scheme and introduces episodic equilibrium as a solution concept for stochastic games (SGs), where agents strategize based on the current state and episode stage. The paper also establishes an upper bound on the approximation error that decays with the episode length for both discounted and time-averaged utilities. This approach bridges the gap in the analysis of finite and infinite-horizon SGs, and provides a unifying framework to address time-averaged and discounted utilities. To show the effectiveness of the scheme, the paper presents episodic, decentralized (i.e., payoff-based), and model-free learning dynamics proven to reach (near) episodic equilibrium in broad classes of SGs, including zero-sum, identical-interest and specific general-sum SGs with switching controllers for both time-averaged and discounted utilities.