Finite-Time Analysis of Projected Two-Time-Scale Stochastic Approximation

TL;DR

This paper provides a finite-time convergence analysis for projected two-time-scale stochastic approximation, deriving explicit error bounds and illustrating their application in reinforcement learning.

Contribution

It introduces a detailed finite-time error bound for projected two-time-scale stochastic approximation with explicit constants and separates approximation and statistical errors.

Findings

Explicit mean-square error bounds are derived.

Constants depend on stability margins and coupling invertibility.

Numerical experiments validate theoretical results.

Abstract

We study the finite-time convergence of projected linear two-time-scale stochastic approximation with constant step sizes and Polyak--Ruppert averaging. We establish an explicit mean-square error bound, decomposing it into two interpretable components, an approximation error determined by the constrained subspace and a statistical error decaying at a sublinear rate, with constants expressed through restricted stability margins and a coupling invertibility condition. These constants cleanly separate the effect of subspace choice (approximation errors) from the effect of the averaging horizon (statistical errors). We illustrate our theoretical results through a number of numerical experiments on both synthetic and reinforcement learning problems.