A Bregman Extension of quasi-Newton updates I: An Information Geometrical framework

TL;DR

This paper extends quasi-Newton methods using Bregman divergences within an information geometric framework, providing new update formulas and insights into their invariance and sparsity properties, with connections to statistical algorithms.

Contribution

It introduces Bregman divergence-based quasi-Newton updates, generalizing the classical methods and revealing their geometric and statistical relationships.

Findings

Extended quasi-Newton formulas derived from Bregman divergences.

Invariance properties of the new update formulas analyzed.

Connections between sparse quasi-Newton methods and statistical algorithms like EM and boosting.

Abstract

We study quasi-Newton methods from the viewpoint of information geometry induced associated with Bregman divergences. Fletcher has studied a variational problem which derives the approximate Hessian update formula of the quasi-Newton methods. We point out that the variational problem is identical to optimization of the Kullback-Leibler divergence, which is a discrepancy measure between two probability distributions. The Kullback-Leibler divergence for the multinomial normal distribution corresponds to the objective function Fletcher has considered. We introduce the Bregman divergence as an extension of the Kullback-Leibler divergence, and derive extended quasi-Newton update formulae based on the variational problem with the Bregman divergence. As well as the Kullback-Leibler divergence, the Bregman divergence introduces the information geometrical structure on the set of positive definite matrices. From the geometrical viewpoint, we study the approximation Hessian update, the invariance property of the update formulae, and the sparse quasi-Newton methods. Especially, we point out that the sparse quasi-Newton method is closely related to statistical methods such as the EM-algorithm and the boosting algorithm. Information geometry is useful tool not only to better understand the quasi-Newton methods but also to design new update formulae.