Objective acceleration for unconstrained optimization

TL;DR

This paper introduces O-ACCEL, a novel acceleration method for unconstrained optimization that minimizes an approximation of the objective function on subspaces, showing improved performance over existing methods.

Contribution

The paper proposes O-ACCEL, a new acceleration scheme that minimizes an objective function approximation, and proves its equivalence to the Full Orthogonalization Method for quadratic objectives.

Findings

O-ACCEL outperforms N-GMRES in tests.

O-ACCEL is competitive with L-BFGS and N-CG.

O-ACCEL can be combined with domain-specific optimizers.

Abstract

Acceleration schemes can dramatically improve existing optimization procedures. In most of the work on these schemes, such as nonlinear Generalized Minimal Residual (N-GMRES), acceleration is based on minimizing the $\ell_2$ norm of some target on subspaces of $\mathbb{R}^n$. There are many numerical examples that show how accelerating general purpose and domain-specific optimizers with N-GMRES results in large improvements. We propose a natural modification to N-GMRES, which significantly improves the performance in a testing environment originally used to advocate N-GMRES. Our proposed approach, which we refer to as O-ACCEL (Objective Acceleration), is novel in that it minimizes an approximation to the \emph{objective function} on subspaces of $\mathbb{R}^n$. We prove that O-ACCEL reduces to the Full Orthogonalization Method for linear systems when the objective is quadratic, which differentiates our proposed approach from existing acceleration methods. Comparisons with L-BFGS and N-CG indicate the competitiveness of O-ACCEL. As it can be combined with domain-specific optimizers, it may also be beneficial in areas where L-BFGS or N-CG are not suitable.