HOUDINI: Escaping from Moderately Constrained Saddles

TL;DR

This paper presents the first polynomial-time algorithms for escaping high-dimensional saddle points in constrained optimization problems, using gradient-based methods under a moderate number of inequality constraints.

Contribution

It introduces novel algorithms that efficiently escape saddle points in constrained settings, extending previous unconstrained results to inequality-constrained problems.

Findings

Algorithms work with a logarithmic number of constraints.

Applicable to both regular and stochastic gradient descent.

First polynomial-time solutions for constrained saddle point escape.

Abstract

We give the first polynomial time algorithms for escaping from high-dimensional saddle points under a moderate number of constraints. Given gradient access to a smooth function $f \colon \mathbb R^d \to \mathbb R$ we show that (noisy) gradient descent methods can escape from saddle points under a logarithmic number of inequality constraints. This constitutes the first tangible progress (without reliance on NP-oracles or altering the definitions to only account for certain constraints) on the main open question of the breakthrough work of Ge et al. who showed an analogous result for unconstrained and equality-constrained problems. Our results hold for both regular and stochastic gradient descent.