Escaping Saddle Points in Constrained Optimization

TL;DR

This paper introduces a framework for escaping saddle points in constrained nonconvex optimization, achieving convergence to second-order stationary points under certain conditions on the convex set and quadratic subproblems.

Contribution

It proposes a generic method for escaping saddle points in constrained problems, with convergence guarantees based on approximate quadratic solutions and extends to stochastic settings.

Findings

Converges to second-order stationary points within a polynomial number of iterations.

Requires solving approximate quadratic programs efficiently for convergence.

Extends results to stochastic optimization with gradient and Hessian evaluations.

Abstract

In this paper, we study the problem of escaping from saddle points in smooth nonconvex optimization problems subject to a convex set $\mathcal{C}$. We propose a generic framework that yields convergence to a second-order stationary point of the problem, if the convex set $\mathcal{C}$ is simple for a quadratic objective function. Specifically, our results hold if one can find a $\rho$-approximate solution of a quadratic program subject to $\mathcal{C}$ in polynomial time, where $\rho<1$ is a positive constant that depends on the structure of the set $\mathcal{C}$. Under this condition, we show that the sequence of iterates generated by the proposed framework reaches an $(\epsilon,\gamma)$-second order stationary point (SOSP) in at most $\mathcal{O}(\max\{\epsilon^{-2},\rho^{-3}\gamma^{-3}\})$ iterations. We further characterize the overall complexity of reaching an SOSP when the convex set $\mathcal{C}$ can be written as a set of quadratic constraints and the objective function Hessian has a specific structure over the convex set $\mathcal{C}$. Finally, we extend our results to the stochastic setting and characterize the number of stochastic gradient and Hessian evaluations to reach an $(\epsilon,\gamma)$-SOSP.