TRSVR: An Adaptive Stochastic Trust-Region Method with Variance Reduction

TL;DR

This paper introduces TRSVR, a stochastic trust-region method with variance reduction for nonconvex optimization, achieving faster convergence without function evaluations, and demonstrating superior performance over SGD and Adam in experiments.

Contribution

The paper presents a novel stochastic trust-region algorithm that incorporates variance reduction, enabling efficient convergence in nonconvex optimization without function value computations.

Findings

Converges to a first-order stationary point under mild assumptions.

Achieves iteration and sample complexity bounds comparable to SVRG-based methods.

Outperforms SGD and Adam in machine learning tasks.

Abstract

We propose a stochastic trust-region method for unconstrained nonconvex optimization that incorporates stochastic variance-reduced gradients (SVRG) to accelerate convergence. Unlike classical trust-region methods, the proposed algorithm relies solely on stochastic gradient information and does not require function value evaluations. The trust-region radius is adaptively adjusted based on a radius-control parameter and the stochastic gradient estimate. Under mild assumptions, we establish that the algorithm converges in expectation to a first-order stationary point. Moreover, the method achieves iteration and sample complexity bounds that match those of SVRG-based first-order methods, while allowing stochastic and potentially gradient-dependent second-order information. Extensive numerical experiments demonstrate that incorporating SVRG accelerates convergence, and that the use of trust-region methods and Hessian information further improves performance. We also highlight the impact of batch size and inner-loop length on efficiency, and show that the proposed method outperforms SGD and Adam on several machine learning tasks.