An investigation of stochastic trust-region based algorithms for finite-sum minimization

TL;DR

This paper analyzes the TRish stochastic optimization algorithm for finite-sum minimization, providing theoretical insights, hyper-parameter tuning strategies, and experimental validation that demonstrate improved performance and robustness.

Contribution

It offers a theoretical analysis of TRish, investigates hyper-parameter tuning, and introduces a practical implementation with gradient estimation tests that enhance performance.

Findings

Theoretical analysis complements existing literature.

Practical implementation improves performance.

Reduces sensitivity to hyper-parameters.

Abstract

This work elaborates on the TRust-region-ish (TRish) algorithm, a stochastic optimization method for finite-sum minimization problems proposed by Curtis et al. in [Curtis2019, Curtis2022]. A theoretical analysis that complements the results in the literature is presented, and the issue of tuning the involved hyper-parameters is investigated. Our study also focuses on a practical version of the method, which computes the stochastic gradient by means of the inner product test and the orthogonality test proposed by Bollapragada et al. in [Bollapragada2018]. It is shown experimentally that this implementation improves the performance of TRish and reduces its sensitivity to the choice of the hyper-parameters.