Gaussian Approximation for Two-Timescale Linear Stochastic Approximation

TL;DR

This paper provides non-asymptotic bounds for the accuracy of normal approximation in two-timescale linear stochastic approximation algorithms, revealing how timescale separation affects convergence in different regimes.

Contribution

It introduces non-asymptotic bounds for normal approximation accuracy in TTSA algorithms, analyzing the impact of timescale separation on convergence rates.

Findings

Normal approximation rate improves with increased timescale separation for the last iterate.

Normal approximation rate decreases with increased timescale separation in Polyak-Ruppert averaging.

High-order moment bounds for TTSA error are established.

Abstract

In this paper, we establish non-asymptotic bounds for accuracy of normal approximation for linear two-timescale stochastic approximation (TTSA) algorithms driven by martingale difference or Markov noise. Focusing on both the last iterate and Polyak-Ruppert averaging regimes, we derive bounds for normal approximation in terms of the convex distance between probability distributions. Our analysis reveals a non-trivial interaction between the fast and slow timescales: the normal approximation rate for the last iterate improves as the timescale separation increases, while it decreases in the Polyak-Ruppert averaged setting. We also provide the high-order moment bounds for the error of linear TTSA algorithm, which may be of independent interest.