Robust Accelerated Adaptive Search: High-Probability Complexity Bounds under Bounded-Moment Stochastic Oracles

TL;DR

This paper introduces RAAS, a robust adaptive search algorithm with momentum for stochastic convex optimization under heavy-tailed noise, providing high-probability complexity bounds.

Contribution

It develops a high-probability framework for adaptive methods with momentum in noisy environments and proposes RAAS with tunable momentum for improved stability and acceleration.

Findings

RAAS achieves high-probability complexity bounds in stochastic convex optimization.

Theoretical analysis clarifies parameter trade-offs between acceleration and stability.

Empirical results show the effectiveness of the switching heuristic.

Abstract

We study unconstrained smooth convex optimization under stochastic first- and zeroth-order oracles subject only to finite-moment bounds, naturally admitting persistent bias and heavy-tailed noise. In this hostile environment, integrating momentum into \emph{adaptive step search} to secure acceleration poses an inherent structural challenge, because momentum propagates oracle errors across iterations, inevitably undermining the stabilizing effect of local search. To address this difficulty, we propose \texttt{RAAS}, a robust accelerated adaptive search method with tunable momentum intervention. Theoretically, we develop a general high-probability framework for adaptive search methods under stochastic oracle feedback, and instantiate it through the strongly convex and general convex analyses of \texttt{RAAS}. This yields high-probability stopping-time complexity bounds for reaching the attainable precision neighborhood. The resulting guarantees also clarify how the algorithmic parameters trade off early-stage acceleration against late-stage stability, and motivate a simple switching heuristic that performs well empirically.