Decentralized Optimal Equilibrium Learning in Stochastic Games via Single-bit Feedback

TL;DR

This paper introduces a decentralized learning method for stochastic games that uses minimal feedback to coordinate agents on welfare-optimizing equilibria, with proven finite-time regret guarantees.

Contribution

It proposes a novel single-bit feedback signaling mechanism for decentralized equilibrium selection that aligns with social welfare objectives in stochastic games.

Findings

Achieves logarithmic expected regret under mild conditions.

Develops explore-and-commit and online algorithms for general stochastic games.

Handles heterogeneous model-based and model-free methods.

Abstract

We study decentralized equilibrium selection in stochastic games under severe information and communication constraints. In such settings, convergence to equilibrium alone is insufficient, as stochastic games typically admit many equilibria with markedly different welfare properties. We address decentralized optimal equilibrium selection, where agents coordinate on equilibria that optimize a designer-specified social welfare objective while allowing heterogeneous tolerance to deviations from strict best responses. Agents observe only the global state trajectory and their realized rewards, and exchange a single randomized bit of feedback per agent per round. This semantic content/discontent signaling mechanism implicitly aligns decentralized learning dynamics with the global welfare objective. We develop explore-and-commit and online variants applicable to general stochastic games, accommodating heterogeneous model-based or model-free methods for solving the induced Markov decision processes, and establish explicit finite-time regret guarantees, showing logarithmic expected regret under mild conditions.