Comparing BFGS and OGR for Second-Order Optimization

TL;DR

This paper compares BFGS and a new method called Online Gradient Regression (OGR) for second-order optimization, highlighting OGR's advantages in non-convex problems and its online gradient-based Hessian estimation.

Contribution

The paper introduces OGR, a novel online gradient regression method for Hessian estimation that handles non-convexity and improves convergence over BFGS.

Findings

OGR converges faster than BFGS in experiments.

OGR effectively estimates non-positive definite Hessians.

Both methods perform well on standard test functions.

Abstract

Estimating the Hessian matrix, especially for neural network training, is a challenging problem due to high dimensionality and cost. In this work, we compare the classical Sherman-Morrison update used in the popular BFGS method (Broy-den-Fletcher-Goldfarb-Shanno), which maintains a positive definite Hessian approximation under a convexity assumption, with a novel approach called Online Gradient Regression (OGR). OGR performs regression of gradients against positions using an exponential moving average to estimate second derivatives online, without requiring Hessian inversion. Unlike BFGS, OGR allows estimation of a general (not necessarily positive definite) Hessian and can thus handle non-convex structures. We evaluate both methods across standard test functions and demonstrate that OGR achieves faster convergence and improved loss, particularly in non-convex settings.