Q-Learning for Stochastic Control under General Information Structures and Non-Markovian Environments

TL;DR

This paper establishes a convergence theorem for Q-learning in complex, non-Markovian stochastic environments, broadening the understanding of reinforcement learning under general conditions and diverse applications.

Contribution

It provides the first convergence theorem for Q-learning in non-Markovian environments with detailed conditions and explores multiple applications in stochastic control and multi-agent systems.

Findings

Convergence conditions involve ergodicity and positivity.

Characterization of the limit of Q-learning iterates.

Applications to POMDPs, belief-MDPs, and multi-agent models.

Abstract

As a primary contribution, we present a convergence theorem for stochastic iterations, and in particular, Q-learning iterates, under a general, possibly non-Markovian, stochastic environment. Our conditions for convergence involve an ergodicity and a positivity criterion. We provide a precise characterization on the limit of the iterates and conditions on the environment and initializations for convergence. As our second contribution, we discuss the implications and applications of this theorem to a variety of stochastic control problems with non-Markovian environments involving (i) quantized approximations of fully observed Markov Decision Processes (MDPs) with continuous spaces (where quantization break down the Markovian structure), (ii) quantized approximations of belief-MDP reduced partially observable MDPS (POMDPs) with weak Feller continuity and a mild version of filter stability (which requires the knowledge of the model by the controller), (iii) finite window approximations of POMDPs under a uniform controlled filter stability (which does not require the knowledge of the model), and (iv) for multi-agent models where convergence of learning dynamics to a new class of equilibria, subjective Q-learning equilibria, will be studied. In addition to the convergence theorem, some implications of the theorem above are new to the literature and others are interpreted as applications of the convergence theorem. Some open problems are noted.