Compact representations of structured BFGS matrices

TL;DR

This paper develops compact matrix representations for structured BFGS updates, enabling efficient large-scale optimization by leveraging problem structure and second-derivative information.

Contribution

It introduces new compact representations for structured BFGS matrices and demonstrates their effectiveness in limited memory algorithms.

Findings

Efficient limited memory algorithms developed

Successful application to large-scale imaging problems

Improved optimization performance observed

Abstract

For general large-scale optimization problems compact representations exist in which recursive quasi-Newton update formulas are represented as compact matrix factorizations. For problems in which the objective function contains additional structure, so-called structured quasi-Newton methods exploit available second-derivative information and approximate unavailable second derivatives. This article develops the compact representations of two structured Broyden-Fletcher-Goldfarb-Shanno update formulas. The compact representations enable efficient limited memory and initialization strategies. Two limited memory line search algorithms are described and tested on a collection of problems, including a real world large scale imaging application.