Convergence guarantees for a class of non-convex and non-smooth optimization problems

TL;DR

This paper provides convergence guarantees for gradient-based methods applied to broad classes of non-convex, non-smooth optimization problems, including faster rates for sub-analytic functions and escape from saddle points.

Contribution

It introduces convergence analysis for three gradient-based methods on non-convex, non-smooth problems, and simplifies the CCCP algorithm while maintaining its convergence properties.

Findings

Established convergence rates for gradient descent, proximal, and Frank-Wolfe methods.

Proved faster convergence for sub-analytic functions.

Showed algorithms can escape strict saddle points in non-smooth settings.

Abstract

We consider the problem of finding critical points of functions that are non-convex and non-smooth. Studying a fairly broad class of such problems, we analyze the behavior of three gradient-based methods (gradient descent, proximal update, and Frank-Wolfe update). For each of these methods, we establish rates of convergence for general problems, and also prove faster rates for continuous sub-analytic functions. We also show that our algorithms can escape strict saddle points for a class of non-smooth functions, thereby generalizing known results for smooth functions. Our analysis leads to a simplification of the popular CCCP algorithm, used for optimizing functions that can be written as a difference of two convex functions. Our simplified algorithm retains all the convergence properties of CCCP, along with a significantly lower cost per iteration. We illustrate our methods and theory via applications to the problems of best subset selection, robust estimation, mixture density estimation, and shape-from-shading reconstruction.