Sign-Separated Finite-Time Error Analysis of Q-Learning

TL;DR

This paper introduces a novel sign-separated finite-time error analysis for Q-learning, revealing an asymmetry in error dynamics linked to overestimation, with bounds applicable to deterministic and stochastic cases.

Contribution

It develops a new analytical framework decomposing Q-learning error into positive and negative parts, highlighting asymmetries and providing finite-time bounds.

Findings

Negative error dynamics are dominated by a stable LTI system.

Positive errors can propagate and cause overestimation.

Finite-time bounds are established for both deterministic and stochastic recursions.

Abstract

This paper develops a sign-separated finite-time error analysis for constant step-size Q-learning. Starting from the switching-system representation, the error is decomposed into its componentwise negative and positive parts. The negative part is dominated by a lower comparison linear time-invariant (LTI) system associated with a fixed optimal policy, whereas the positive part is controlled by a linear switching system. The resulting bounds show that the negative-side LTI certificate is no slower than the positive-side switching certificate and may produce a faster exponential envelope. The analysis identifies a max-induced asymmetry in Q-learning error dynamics. This asymmetry is connected to overestimation: positive action-wise errors can be selected and propagated by the Bellman maximum, whereas negative errors admit an optimal-policy lower comparison. Finite-time bounds are provided for both deterministic and stochastic constant-step-size recursions.