Adaptive Variance Reduction for Stochastic Optimization under Weaker Assumptions

TL;DR

This paper introduces a new adaptive variance reduction method for stochastic optimization that achieves optimal convergence rates under weaker assumptions, extending to compositional and finite-sum problems.

Contribution

The paper proposes a novel adaptive STORM algorithm with weaker assumptions and optimal convergence rates, also extending to compositional and finite-sum optimization.

Findings

Achieves $oldsymbol{ ext{O}(T^{-1/3})}$ convergence for non-convex functions.

Requires weaker assumptions than previous methods.

Validates effectiveness through numerical experiments.

Abstract

This paper explores adaptive variance reduction methods for stochastic optimization based on the STORM technique. Existing adaptive extensions of STORM rely on strong assumptions like bounded gradients and bounded function values, or suffer an additional $\mathcal{O}(\log T)$ term in the convergence rate. To address these limitations, we introduce a novel adaptive STORM method that achieves an optimal convergence rate of $\mathcal{O}(T^{-1/3})$ for non-convex functions with our newly designed learning rate strategy. Compared with existing approaches, our method requires weaker assumptions and attains the optimal convergence rate without the additional $\mathcal{O}(\log T)$ term. We also extend the proposed technique to stochastic compositional optimization, obtaining the same optimal rate of $\mathcal{O}(T^{-1/3})$. Furthermore, we investigate the non-convex finite-sum problem and develop another innovative adaptive variance reduction method that achieves an optimal convergence rate of $\mathcal{O}(n^{1/4} T^{-1/2} )$, where $n$ represents the number of component functions. Numerical experiments across various tasks validate the effectiveness of our method.