Full Convergence of Regularized Methods for Unconstrained Optimization

TL;DR

This paper proves that a broad class of unconstrained optimization algorithms with regularized quadratic models, using only function and gradient evaluations, converge globally under pseudoconvexity assumptions.

Contribution

It extends convergence results to a wider class of regularized methods with variable metrics, beyond convex functions.

Findings

Whole sequence convergence under pseudoconvexity

Use of variable metric quasi-Fejér monotonicity

Applicable to algorithms with local quadratic models

Abstract

Typically, the sequence of points generated by an optimization algorithm may have multiple limit points. Under convexity assumptions, however, (sub)gradient methods are known to generate a convergent sequence of points. In this paper, we extend the latter property to a broader class of algorithms. Specifically, we study unconstrained optimization methods that use local quadratic models regularized by a power $r \ge 3$ of the norm of the step. In particular, we focus on the case where only the objective function and its gradient are evaluated. Our analysis shows that, by a careful choice of the regularized model at every iteration, the whole sequence of points generated by this class of algorithms converges if the objective function is pseudoconvex. The result is achieved by employing appropriate matrices to ensure that the sequence of points is variable metric quasi-Fej\'er monotone.