Quasi-Newton Methods: A New Direction

TL;DR

This paper reinterprets quasi-Newton methods as Bayesian linear regression approximations, revealing their limitations and proposing a novel nonparametric approach that improves efficiency while maintaining similar computational costs.

Contribution

It introduces a Bayesian perspective to quasi-Newton methods and develops a new nonparametric algorithm for better optimization performance.

Findings

Classical quasi-Newton methods can be viewed as Bayesian linear regression.

The new nonparametric quasi-Newton method outperforms traditional algorithms.

The approach offers more efficient use of information with comparable computational costs.

Abstract

Four decades after their invention, quasi-Newton methods are still state of the art in unconstrained numerical optimization. Although not usually interpreted thus, these are learning algorithms that fit a local quadratic approximation to the objective function. We show that many, including the most popular, quasi-Newton methods can be interpreted as approximations of Bayesian linear regression under varying prior assumptions. This new notion elucidates some shortcomings of classical algorithms, and lights the way to a novel nonparametric quasi-Newton method, which is able to make more efficient use of available information at computational cost similar to its predecessors.