Finite-time analysis of Multi-timescale Stochastic Optimization Algorithms

TL;DR

This paper provides finite-time convergence guarantees for two multi-timescale zeroth-order stochastic optimization algorithms, including gradient and Newton-based methods, with theoretical analysis and experimental validation.

Contribution

It offers the first finite-time analysis for multi-timescale zeroth-order stochastic algorithms, including error bounds and step-size strategies.

Findings

Derived mean-squared error bounds for Hessian estimation.

Established convergence rates to first-order stationary points.

Validated theoretical results with experiments on the Continuous Mountain Car environment.

Abstract

We present a finite-time analysis of two smoothed functional stochastic approximation algorithms for simulation-based optimization. The first is a two time-scale gradient-based method, while the second is a three time-scale Newton-based algorithm that estimates both the gradient and the Hessian of the objective function $J$. Both algorithms involve zeroth order estimates for the gradient/Hessian. Although the asymptotic convergence of these algorithms has been established in prior work, finite-time guarantees of two-timescale stochastic optimization algorithms in zeroth order settings have not been provided previously. For our Newton algorithm, we derive mean-squared error bounds for the Hessian estimator and establish a finite-time bound on $\min\limits_{0 \le m \le T} \mathbb{E}\| \nabla J(\theta(m)) \|^2$, showing convergence to first-order stationary points. The analysis explicitly characterizes the interaction between multiple time-scales and the propagation of estimation errors. We further identify step-size choices that balance dominant error terms and achieve near-optimal convergence rates. We also provide corresponding finite-time guarantees for the gradient algorithm under the same framework. The theoretical results are further validated through experiments on the Continuous Mountain Car environment.