Nonlinear Two-Time-Scale Stochastic Approximation: Convergence and Finite-Time Performance

TL;DR

This paper analyzes the convergence and finite-time performance of nonlinear two-time-scale stochastic approximation, providing new theoretical guarantees and convergence rates that extend beyond the linear case.

Contribution

It offers the first finite-time convergence analysis for nonlinear two-time-scale stochastic approximation under standard assumptions.

Findings

Convergence rate of (1/k^{2/3}) in expectation.

Characterization of the coupling between fast and slow iterates.

Finite-time bounds for nonlinear stochastic approximation.

Abstract

Two-time-scale stochastic approximation, a generalized version of the popular stochastic approximation, has found broad applications in many areas including stochastic control, optimization, and machine learning. Despite its popularity, theoretical guarantees of this method, especially its finite-time performance, are mostly achieved for the linear case while the results for the nonlinear counterpart are very sparse. Motivated by the classic control theory for singularly perturbed systems, we study in this paper the asymptotic convergence and finite-time analysis of the nonlinear two-time-scale stochastic approximation. Under some fairly standard assumptions, we provide a formula that characterizes the rate of convergence of the main iterates to the desired solutions. In particular, we show that the method achieves a convergence in expectation at a rate $\mathcal{O}(1/k^{2/3})$, where $k$ is the number of iterations. The key idea in our analysis is to properly choose the two step sizes to characterize the coupling between the fast and slow-time-scale iterates.