Quasi-Newton method of Optimization is proved to be a steepest descent method under the ellipsoid norm

TL;DR

This paper proves that Quasi-Newton methods can be viewed as steepest descent methods when measured with an ellipsoid norm, providing new theoretical insights into their convergence properties.

Contribution

The paper demonstrates that Quasi-Newton methods are equivalent to steepest descent methods under an ellipsoid norm, generalizing classical inequalities and deepening theoretical understanding.

Findings

Quasi-Newton methods are steepest descent under ellipsoid norm

Introduction of generalized Cauchy-Schwartz inequalities

Theoretical proof of equivalence between Quasi-Newton and steepest descent

Abstract

Optimization problems, arise in many practical applications, from the view points of both theory and numerical methods. Especially, significant improvement in deep learning training came from the Quasi-Newton methods. Quasi-Newton search directions provide an attractive alternative to Newton's method in that they do not require computation of the Hessian and yet still attain a super linear rate of convergence. In Quasi-Newton method, we require Hessian approximation to satisfy the secant equation. In this paper, the Classical Cauchy-Schwartz Inequality is introduced, then more generalization are proposed. And it is seriously proved that Quasi-Newton method is a steepest descent method under the ellipsoid norm.