Skip the Hessian, Keep the Rates: Globalized Semismooth Newton with Lazy Hessian Updates

TL;DR

This paper introduces a semismooth Newton method that reduces computational costs by lazy Hessian updates, achieving fast convergence in nonsmooth optimization problems relevant to machine learning.

Contribution

It proposes a novel semismooth Newton algorithm with lazy Hessian updates, providing global convergence and superlinear local convergence without requiring second-order differentiability.

Findings

Method achieves significant speedups in experiments

Maintains strong convergence guarantees

Effective for nonsmooth ML optimization problems

Abstract

Second-order methods are provably faster than first-order methods, and their efficient implementations for large-scale optimization problems have attracted significant attention. Yet, optimization problems in ML often have nonsmooth derivatives, which makes the existing convergence rate theory of second-order methods inapplicable. In this paper, we propose a new semismooth Newton method (SSN) that enjoys both global convergence rates and asymptotic superlinear convergence without requiring second-order differentiability. Crucially, our method does not require (generalized) Hessians to be evaluated at each iteration but only periodically, and it reuses stale Hessians otherwise (i.e., it performs lazy Hessian updates), saving compute cost and often leading to significant speedups in time, whilst still maintaining strong global and local convergence rate guarantees. We develop our theory in an infinite-dimensional setting and illustrate it with numerical experiments on matrix factorization and neural networks with Lipschitz constraints.