A Dense Initialization for Limited-Memory Quasi-Newton Methods

TL;DR

This paper introduces a novel dense eigendecomposition-based initialization for limited-memory quasi-Newton methods, improving performance in nonconvex optimization by leveraging a shape-changing trust-region approach.

Contribution

It proposes a new dense initialization scheme for L-BFGS that enhances optimization efficiency without explicitly forming matrices, applicable to various quasi-Newton methods.

Findings

Outperforms traditional diagonal initialization in numerical tests

Effective in nonconvex unconstrained optimization problems

Broad applicability to quasi-Newton trust-region and line search methods

Abstract

We consider a family of dense initializations for limited-memory quasi-Newton methods. The proposed initialization exploits an eigendecomposition-based separation of the full space into two complementary subspaces, assigning a different initialization parameter to each subspace. This family of dense initializations is proposed in the context of a limited-memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) trust-region method that makes use of a shape-changing norm to define each subproblem. As with L-BFGS methods that traditionally use diagonal initialization, the dense initialization and the sequence of generated quasi-Newton matrices are never explicitly formed. Numerical experiments on the CUTEst test set suggest that this initialization together with the shape-changing trust-region method outperforms other L-BFGS methods for solving general nonconvex unconstrained optimization problems. While this dense initialization is proposed in the context of a special trust-region method, it has broad applications for more general quasi-Newton trust-region and line search methods. In fact, this initialization is suitable for use with any quasi-Newton update that admits a compact representation and, in particular, any member of the Broyden class of updates.