Second-order methods for provably escaping strict saddle points in composite nonconvex and nonsmooth optimization

TL;DR

This paper presents two novel second-order algorithms that reliably escape strict saddle points in complex nonconvex, nonsmooth optimization problems, backed by theoretical guarantees and preliminary numerical validation.

Contribution

Introduction of the first second-order methods with provable escape from nonsmooth strict saddle points in composite nonconvex optimization.

Findings

Algorithms converge to second-order stationary points.

The methods are effective regardless of initialization.

Preliminary experiments show promising performance.

Abstract

This study introduces two second-order methods designed to provably avoid saddle points in composite nonconvex optimization problems: (i) a nonsmooth trust-region method and (ii) a curvilinear linesearch method. These developments are grounded in the forward-backward envelope (FBE), for which we analyze the local second-order differentiability around critical points and establish a novel equivalence between its second-order stationary points and those of the original objective. We show that the proposed algorithms converge to second-order stationary points of the FBE under a mild local smoothness condition on the proximal mapping of the nonsmooth term. Notably, for \( \C^2 \)-partly smooth functions, this condition holds under a standard strict complementarity assumption. To the best of our knowledge, these are the first second-order algorithms that provably escape nonsmooth strict saddle points of composite nonconvex optimization, regardless of the initialization. Our preliminary numerical experiments show promising performance of the developed methods, validating our theoretical foundations.