Adaptive Regularization Minimization Algorithms with Non-Smooth Norms and Euclidean Curvature

TL;DR

This paper introduces adaptive regularization algorithms for unconstrained nonlinear minimization that effectively handle non-smooth norms, providing complexity bounds and new optimality conditions, applicable to both first- and second-order approximate solutions.

Contribution

It extends the AR1pGN algorithm to non-smooth norms, establishes evaluation complexity bounds independent of norm equivalence, and develops new optimality conditions for quadratic regularized problems.

Findings

Evaluation complexity bounds are unaffected by non-smoothness.

The $O(\epsilon_2^{-3})$ bound holds for second-order minimizers.

New optimality conditions for quadratic regularized problems are proposed.

Abstract

A regularization algorithm (AR1pGN) for unconstrained nonlinear minimization is considered, which uses a model consisting of a Taylor expansion of arbitrary degree and regularization term involving a possibly non-smooth norm. It is shown that the non-smoothness of the norm does not affect the $O(\epsilon_1^{-(p+1)/p})$ upper bound on evaluation complexity for finding first-order $\epsilon_1$-approximate minimizers using $p$ derivatives, and that this result does not hinge on the equivalence of norms in $\Re^n$. It is also shown that, if $p=2$, the bound of $O(\epsilon_2^{-3})$ evaluations for finding second-order $\epsilon_2$-approximate minimizers still holds for a variant of AR1pGN named AR2GN, despite the possibly non-smooth nature of the regularization term. Moreover, the adaptation of the existing theory for handling the non-smoothness results in an interesting modification of the subproblem termination rules, leading to an even more compact complexity analysis. In particular, it is shown when the Newton's step is acceptable for an adaptive regularization method. The approximate minimization of quadratic polynomials regularized with non-smooth norms is then discussed, and a new approximate second-order necessary optimality condition is derived for this case. An specialized algorithm is then proposed to enforce the first- and second-order conditions that are strong enough to ensure the existence of a suitable step in AR1pGN (when $p=2$) and in AR2GN, and its iteration complexity is analyzed.