Finite-Time Analysis and Restarting Scheme for Linear Two-Time-Scale Stochastic Approximation

TL;DR

This paper analyzes the finite-time convergence of linear two-time-scale stochastic approximation methods with time-varying step sizes, introduces a restarting scheme to improve performance, and demonstrates practical benefits for reinforcement learning applications.

Contribution

The paper provides the first finite-time complexity analysis of the method under Markovian noise and proposes a restarting scheme that ensures convergence with practical step size management.

Findings

Mean square error converges at rate O(k^{2/3})

Restarting scheme achieves convergence comparable to constant step sizes

Restarting prevents step sizes from becoming too small in practice

Abstract

Motivated by their broad applications in reinforcement learning, we study the linear two-time-scale stochastic approximation, an iterative method using two different step sizes for finding the solutions of a system of two equations. Our main focus is to characterize the finite-time complexity of this method under time-varying step sizes and Markovian noise. In particular, we show that the mean square errors of the variables generated by the method converge to zero at a sublinear rate $\Ocal(k^{2/3})$, where $k$ is the number of iterations. We then improve the performance of this method by considering the restarting scheme, where we restart the algorithm after every predetermined number of iterations. We show that using this restarting method the complexity of the algorithm under time-varying step sizes is as good as the one using constant step sizes, but still achieving an exact converge to the desired solution. Moreover, the restarting scheme also helps to prevent the step sizes from getting too small, which is useful for the practical implementation of the linear two-time-scale stochastic approximation.