Quadratic and Cubic Regularisation Methods with Inexact function and Random Derivatives for Finite-Sum Minimisation

TL;DR

This paper introduces regularisation methods up to third order with inexact function and derivative evaluations for finite-sum minimisation, achieving optimal complexity bounds and demonstrating preliminary numerical results in nonconvex binary classification.

Contribution

It develops a novel framework for high-order regularisation methods with inexact evaluations, providing complexity guarantees matching deterministic optimal bounds.

Findings

Expected iteration complexity matches deterministic bounds.

Preliminary numerical tests in nonconvex binary classification.

Methods effectively handle inexact function and derivative evaluations.

Abstract

This paper focuses on regularisation methods using models up to the third order to search for up to second-order critical points of a finite-sum minimisation problem. The variant presented belongs to the framework of [3]: it employs random models with accuracy guaranteed with a sufficiently large prefixed probability and deterministic inexact function evaluations within a prescribed level of accuracy. Without assuming unbiased estimators, the expected number of iterations is $\mathcal{O}\bigl(\epsilon_1^{-2}\bigr)$ or $\mathcal{O}\bigl(\epsilon_1^{-{3/2}}\bigr)$ when searching for a first-order critical point using a second or third order model, respectively, and of $\mathcal{O}\bigl(\max[\epsilon_1^{-{3/2}},\epsilon_2^{-3}]\bigr)$ when seeking for second-order critical points with a third order model, in which $\epsilon_j$, $j\in\{1,2\}$, is the $j$th-order tolerance. These results match the worst-case optimal complexity for the deterministic counterpart of the method. Preliminary numerical tests for first-order optimality in the context of nonconvex binary classification in imaging, with and without Artifical Neural Networks (ANNs), are presented and discussed.