Examples of slow convergence for adaptive regularization optimization methods are not isolated

TL;DR

This paper demonstrates that slow convergence in adaptive regularization methods for nonconvex optimization is not an isolated case but occurs for a broad class of univariate functions, challenging the perception of their typical performance.

Contribution

The paper shows that slow convergence examples are not isolated and occur for a subset of univariate functions with nonzero measure, expanding understanding of algorithm behavior.

Findings

Slow convergence examples are not isolated.

Slow convergence occurs for a subset of univariate functions.

Behavior affects a set of functions with nonzero measure.

Abstract

The adaptive regularization algorithm for unconstrained nonconvex optimization was shown in Nesterov and Polyak (2006) and Cartis, Gould and Toint (2011) to require, under standard assumptions, at most $\mathcal{O}(\epsilon^{3/(3-q)})$ evaluations of the objective function and its derivatives of degrees one and two to produce an $\epsilon$-approximate critical point of order $q\in\{1,2\}$. This bound was shown to be sharp for $q \in\{1,2\}$. This note revisits these results and shows that the example for which slow convergence is exhibited is not isolated, but that this behaviour occurs for a subset of univariate functions of nonzero measure.