Tail bounds for stochastic approximation

TL;DR

This paper analyzes the probabilistic tail bounds for stochastic approximation methods, showing conditions under which their convergence closely matches deterministic gradient methods with high probability.

Contribution

It provides new probabilistic bounds and conditions ensuring stochastic approximation methods achieve near-deterministic convergence rates with high probability.

Findings

Established tail bounds for stochastic approximation algorithms.

Identified conditions for high-probability convergence guarantees.

Enhanced understanding of variance reduction effects in stochastic methods.

Abstract

Stochastic-approximation gradient methods are attractive for large-scale convex optimization because they offer inexpensive iterations. They are especially popular in data-fitting and machine-learning applications where the data arrives in a continuous stream, or it is necessary to minimize large sums of functions. It is known that by appropriately decreasing the variance of the error at each iteration, the expected rate of convergence matches that of the underlying deterministic gradient method. Conditions are given under which this happens with overwhelming probability.