A Lyapunov Optimization Approach to Repeated Stochastic Games

TL;DR

This paper introduces a Lyapunov optimization framework for repeated stochastic games, enabling online decision-making to maximize utilities under equilibrium constraints without prior probability knowledge.

Contribution

It develops a novel Lyapunov-based online algorithm for stochastic games that achieves equilibrium constraints and maximizes utility functions without needing probability distributions.

Findings

Algorithm converges in polynomial time.

Players are incentivized to participate through equilibrium constraints.

Method can compute correlated equilibrium with higher complexity.

Abstract

This paper considers a time-varying game with $N$ players. Every time slot, players observe their own random events and then take a control action. The events and control actions affect the individual utilities earned by each player. The goal is to maximize a concave function of time average utilities subject to equilibrium constraints. Specifically, participating players are provided access to a common source of randomness from which they can optimally correlate their decisions. The equilibrium constraints incentivize participation by ensuring that players cannot earn more utility if they choose not to participate. This form of equilibrium is similar to the notions of Nash equilibrium and correlated equilibrium, but is simpler to attain. A Lyapunov method is developed that solves the problem in an online \emph{max-weight} fashion by selecting actions based on a set of time-varying weights. The algorithm does not require knowledge of the event probabilities and has polynomial convergence time. A similar method can be used to compute a standard correlated equilibrium, albeit with increased complexity.