Adaptive Regularization within Trust Region Methods for Stochastic Nonconvex Optimization

TL;DR

This paper introduces Reg-ASTRO, an adaptive trust-region method for stochastic nonconvex optimization, achieving improved iteration and sample complexities under broad noise conditions.

Contribution

It develops a novel adaptive regularized trust-region algorithm with decision-dependent noise handling, advancing theoretical convergence guarantees for stochastic nonconvex problems.

Findings

Achieves almost sure rac{1}{1.5} ext{ iteration complexity}

Establishes rac{1}{4.5} ext{ sample complexity}

Outperforms first-order methods in theory and experiments

Abstract

We propose a stochastic nonconvex optimization algorithm that achieves almost sure $\tilde{\mathcal{O}}(\epsilon^{-1.5})$ iteration complexity for problems with smooth objective functions and gradients only observable with noise. The mean-zero stochastic noise is decision-dependent and has unbounded support with subexponential tail, allowing our framework to cover a broad class of problems. The improved almost sure iteration complexity is achieved with a new variant of the adaptive sampling trust-region optimization (ASTRO) augmented with an adaptively regularized local model, which we term Reg-ASTRO. Adaptive sampling ensures that the estimation precision is aligned with a measure of stationarity, so that iterates closer to stationarity trigger higher accuracy requirement for sampling. A key analytical challenge arises because the trust-region radius and regularization are coupled and not determined prior to gradient estimation at each iteration. We further establish an almost sure $\tilde{\mathcal{O}}(\epsilon^{-4.5})$ sample complexity for Reg-ASTRO, which improves to $\tilde{\mathcal{O}}(\epsilon^{-3.5})$ under stronger regularity conditions and use of common random numbers, substantially outperforming first-order methods in theory and numerical experiments.