Technical results on the convergence of quasi-Newton methods for nonsmooth optimization

TL;DR

This paper investigates the convergence behavior of quasi-Newton methods, especially BFGS, for nonsmooth optimization, providing new theoretical insights based on eigenvalue analysis and numerical experiments.

Contribution

It derives eigenvalue behavior assumptions that could ensure convergence of quasi-Newton methods on nonsmooth functions and explores how these methods navigate piecewise structures.

Findings

Eigenvalues of quasi-Newton matrices tend to vanish in certain conditions.

Assumptions on eigenvalue behavior can imply convergence for nonsmooth functions.

Quasi-Newton methods can effectively explore piecewise differentiable structures.

Abstract

It is well-known by now that the BFGS method is an effective method for minimizing nonsmooth functions. However, despite its popularity, theoretical convergence results are almost non-existent. One of the difficulties when analyzing the nonsmooth case is the fact that the secant equation forces certain eigenvalues of the quasi-Newton matrix to vanish, which is a behavior that has not yet been fully analyzed. In this article, we show what kind of behavior of the eigenvalues would be sufficient to be able to prove the convergence for piecewise differentiable functions. More precisely, we derive assumptions on the behavior from numerical experiments and then prove criticality of the limit under these assumptions. Furthermore, we show how quasi-Newton methods are able to explore the piecewise structure. While we do not prove that the observed behavior of the eigenvalues actually occurs, we believe that these results still give insight, and a certain intuition, for the convergence for nonsmooth functions.