Q-Learning for Continuous State and Action MDPs under Average Cost Criteria

TL;DR

This paper develops discretization methods and Q-learning algorithms for infinite-horizon average-cost MDPs with continuous state and action spaces, providing convergence guarantees and near-optimality results.

Contribution

It introduces new discretization techniques and Q-learning algorithms for continuous spaces, with rigorous convergence analysis and error bounds under weaker continuity assumptions.

Findings

Discretization approximation with error bounds under weak and Wasserstein continuity.

Convergence of synchronous and asynchronous Q-learning algorithms in continuous spaces.

Near-optimality of solutions obtained via quantized Q-learning algorithms.

Abstract

For infinite-horizon average-cost criterion problems, there exist relatively few rigorous approximation and reinforcement learning results. In this paper, for Markov Decision Processes (MDPs) with standard Borel spaces, (i) we first provide a discretization based approximation method for MDPs with continuous spaces under average cost criteria, and provide error bounds for approximations when the dynamics are only weakly continuous (for asymptotic convergence of errors as the grid sizes vanish) or Wasserstein continuous (with a rate in approximation as the grid sizes vanish) under certain ergodicity assumptions. In particular, we relax the total variation condition given in prior work to weak continuity or Wasserstein continuity. (ii) We provide synchronous and asynchronous (quantized) Q-learning algorithms for continuous spaces via quantization (where the quantized state is taken to be the actual state in corresponding Q-learning algorithms presented in the paper), and establish their convergence. (iii) We finally show that the convergence is to the optimal Q values of a finite approximate model constructed via quantization, which implies near optimality of the arrived solution. Our Q-learning convergence results and their convergence to near optimality are new for continuous spaces, and the proof method is new even for finite spaces, to our knowledge.