A Discrete-Time Switching System Analysis of Q-learning

TL;DR

This paper introduces a control-theoretic framework to analyze the finite-time convergence of Q-learning by modeling its dynamics as a discrete-time stochastic switching system, providing new error bounds and insights.

Contribution

It presents a novel control-theoretic approach to analyze Q-learning's convergence, deriving finite-time error bounds and explaining overestimation phenomena.

Findings

Finite-time error bounds for Q-learning with constant stepsize.

Q-learning dynamics modeled as a stochastic affine switching system.

Validation through numerical simulations.

Abstract

This paper develops a novel control-theoretic framework to analyze the non-asymptotic convergence of Q-learning. We show that the dynamics of asynchronous Q-learning with a constant step-size can be naturally formulated as a discrete-time stochastic affine switching system. Moreover, the evolution of the Q-learning estimation error is over- and underestimated by trajectories of two simpler dynamical systems. Based on these two systems, we derive a new finite-time error bound of asynchronous Q-learning when a constant stepsize is used. Our analysis also sheds light on the overestimation phenomenon of Q-learning. We further illustrate and validate the analysis through numerical simulations.