Convergence Rate of Stochastic Gradient Search in the Case of Multiple and Non-Isolated Minima

TL;DR

This paper establishes new bounds on the convergence rate of stochastic gradient algorithms, accommodating multiple non-isolated minima and relaxing previous restrictions, with applications to prediction error and reinforcement learning algorithms.

Contribution

It introduces a novel analysis framework that handles multiple, non-isolated minima without Hessian restrictions, extending convergence rate results to broader classes of stochastic gradient methods.

Findings

Derived tight convergence bounds for stochastic gradient algorithms.

Applied results to recursive prediction error identification.

Analyzed convergence rates in supervised and temporal-difference learning.

Abstract

The convergence rate of stochastic gradient search is analyzed in this paper. Using arguments based on differential geometry and Lojasiewicz inequalities, tight bounds on the convergence rate of general stochastic gradient algorithms are derived. As opposed to the existing results, the results presented in this paper allow the objective function to have multiple, non-isolated minima, impose no restriction on the values of the Hessian (of the objective function) and do not require the algorithm estimates to have a single limit point. Applying these new results, the convergence rate of recursive prediction error identification algorithms is studied. The convergence rate of supervised and temporal-difference learning algorithms is also analyzed using the results derived in the paper.