Markovian Foundations for Quasi-Stochastic Approximation in Two Timescales: Extended Version

TL;DR

This paper extends the quasi-stochastic approximation framework to two timescale algorithms, improving convergence rates and providing new analysis methods, with applications in reinforcement learning and extremum seeking control.

Contribution

It introduces a novel approach to analyze two timescale algorithms as single timescale QSA using negative Lyapunov exponents, enhancing convergence rate bounds.

Findings

Achieved near-optimal MSE bounds of O(n^{-4}) in QSA.

Extended QSA analysis to two timescale algorithms.

Applied theory to extremum seeking control (ESC).

Abstract

Many machine learning and optimization algorithms can be cast as instances of stochastic approximation (SA). The convergence rate of these algorithms is known to be slow, with the optimal mean squared error (MSE) of order $O(n^{-1})$. In prior work it was shown that MSE bounds approaching $O(n^{-4})$ can be achieved through the framework of quasi-stochastic approximation (QSA); essentially SA with careful choice of deterministic exploration. These results are extended to two time-scale algorithms, as found in policy gradient methods of reinforcement learning and extremum seeking control. The extensions are made possible in part by a new approach to analysis, allowing for the interpretation of two timescale algorithms as instances of single timescale QSA, made possible by the theory of negative Lyapunov exponents for QSA. The general theory is illustrated with applications to extremum seeking control (ESC).