Unified High-Probability Analysis of Stochastic Variance-Reduced Estimation

TL;DR

This paper introduces a unified high-probability analysis framework for stochastic variance-reduced estimators, applicable across various optimization methods and spaces, improving confidence bounds and complexity results.

Contribution

It develops a general recursion-based framework for variance reduction, providing a high-probability bound using a new vector-valued Freedman inequality, applicable in Euclidean and non-Euclidean spaces.

Findings

Unified high-probability bound for stochastic estimators

Logarithmic dependence on confidence level in mirror descent

First $ ilde{O}( ext{epsilon}^{-3})$ complexity for constrained stochastic optimization

Abstract

Stochastic estimators are fundamental to large-scale optimization, where population quantities must be inferred from noisy oracle observations. Although influential methods such as momentum, SPIDER, STORM, and PAGE have been highly successful, their analyses are largely estimator-specific and expectation-based, obscuring the structural tradeoffs that determine reliability. In this paper, we develop a unified framework for stochastic variance-reduced estimation based on a recursion with three components: memory retention, reset probability, and a correction term for iterate movement. This framework recovers several classical estimators, motivates new second-order variants, and yields a bias-variance decomposition of estimation error. Our main result is a unified high-probability bound proved using a new dimension-free vector-valued Freedman inequality, valid for smooth normed spaces involving random sums of vector martingales. The result applies in both Euclidean and non-Euclidean settings, including the analysis of mirror-descent-based methods in Banach spaces. As applications, we obtain high-probability oracle complexities for unconstrained optimization with mirror descent, establishing the logarithmic dependence on the confidence level. We also derive the first $\tilde{\mathcal{O}}(\varepsilon^{-3})$ oracle-complexity bounds for stochastic optimization with expectation constraints, improving upon the existing $\tilde{\mathcal{O}}(\varepsilon^{-4})$ complexity by leveraging variance-reduced estimation for the first time in this setting.