On the Heterogeneity of Independent Learning Dynamics in Zero-sum Stochastic Games

TL;DR

This paper investigates the convergence of a two-timescale learning algorithm combining fictitious play and Q-learning in zero-sum stochastic games, providing theoretical guarantees under specific conditions.

Contribution

It introduces a novel Lyapunov function approach and proves almost sure convergence for a two-timescale learning method with player-dependent rates.

Findings

Convergence is guaranteed when the discount factor is below a certain threshold.

The method converges almost surely under standard stochastic approximation assumptions.

A new Lyapunov function formulation is proposed for analyzing convergence.

Abstract

We analyze the convergence properties of the two-timescale fictitious play combining the classical fictitious play with the Q-learning for two-player zero-sum stochastic games with player-dependent learning rates. We show its almost sure convergence under the standard assumptions in two-timescale stochastic approximation methods when the discount factor is less than the product of the ratios of player-dependent step sizes. To this end, we formulate a novel Lyapunov function formulation and present a one-sided asynchronous convergence result.