Fully stochastic trust-region methods with Barzilai-Borwein steplengths

TL;DR

This paper introduces TRishBB, a stochastic trust-region method with Barzilai-Borwein steplengths, improving efficiency and accuracy in nonconvex finite-sum minimization problems without requiring diminishing steps or full gradients.

Contribution

It develops a new stochastic trust-region framework with Barzilai-Borwein steplengths, providing convergence analysis and demonstrating improved performance in machine learning tasks.

Findings

Enhanced testing accuracy over existing methods

Convergence without diminishing step-sizes

Effective in nonconvex optimization scenarios

Abstract

We investigate stochastic gradient methods and stochastic counterparts of the Barzilai-Borwein steplengths and their application to finite-sum minimization problems. Our proposal is based on the Trust-Region-ish (TRish) framework introduced in [F. E. Curtis, K. Scheinberg, R. Shi, {\it A stochastic trust region algorithm based on careful step normalization}, Informs Journal on Optimization, 1, 2019]. The new framework, named TRishBB, aims to enhance the performance of TRish and at reducing the computational cost of the second-order TRish variant. We propose three different methods belonging to the TRishBB framework and present the convergence analysis for possibly nonconvex objective functions, considering biased and unbiased gradient approximations. Our analysis requires neither diminishing step-sizes nor full gradient evaluation. The numerical experiments in machine learning applications demonstrate the effectiveness of applying the Barzilai-Borwein steplength with stochastic gradients and show improved testing accuracy compared to the TRish method.