Decentralized Q-Learning for Stochastic Teams and Games

TL;DR

This paper introduces decentralized Q-learning algorithms for stochastic games, demonstrating their convergence to equilibrium policies in weakly acyclic cases with local information, addressing challenges of learning in multi-agent stochastic environments.

Contribution

It presents novel decentralized Q-learning algorithms for stochastic games and proves their convergence in weakly acyclic scenarios, including team problems, with minimal information requirements.

Findings

Algorithms converge to equilibrium policies almost surely

Decentralized approach requires only local information

Applicable to a broad class of stochastic games

Abstract

There are only a few learning algorithms applicable to stochastic dynamic teams and games which generalize Markov decision processes to decentralized stochastic control problems involving possibly self-interested decision makers. Learning in games is generally difficult because of the non-stationary environment in which each decision maker aims to learn its optimal decisions with minimal information in the presence of the other decision makers who are also learning. In stochastic dynamic games, learning is more challenging because, while learning, the decision makers alter the state of the system and hence the future cost. In this paper, we present decentralized Q-learning algorithms for stochastic games, and study their convergence for the weakly acyclic case which includes team problems as an important special case. The algorithm is decentralized in that each decision maker has access to only its local information, the state information, and the local cost realizations; furthermore, it is completely oblivious to the presence of other decision makers. We show that these algorithms converge to equilibrium policies almost surely in large classes of stochastic games.