Stochastic Adaptive Optimization with Unreliable Inputs: A Unified Framework for High-Probability Complexity Analysis

TL;DR

This paper introduces a unified framework for stochastic optimization that provides high-probability complexity bounds in scenarios with unreliable gradient and function estimates, applicable to real-world problems with outliers.

Contribution

It develops an algorithmic and analytical framework that handles corrupted gradient estimates and heavy-tailed noise, unifying line search and trust region methods.

Findings

Provides high-probability bounds on iteration complexity.

Handles arbitrarily corrupted gradient estimates with probability > 1/2.

Addresses heavy-tailed noise in function value estimates.

Abstract

We consider an unconstrained continuous optimization problem where, in each iteration, gradient estimates may be arbitrarily corrupted with a probability greater than 1/2. Additionally, function value estimates may exhibit heavy-tailed noise. This setting captures challenging scenarios where both gradient and function value estimates can be unreliable, making it applicable to many real-world problems, which can have outliers and data anomalies. We introduce an algorithmic and analytical framework that provides high-probability bounds on iteration complexity for this setting. The analysis offers a unified approach, encompassing methods such as line search and trust region.