Lower Bounds for Finding Stationary Points I

TL;DR

This paper establishes tight lower bounds on the number of queries needed by algorithms to find approximate stationary points in smooth, possibly non-convex functions, demonstrating the optimality of several common optimization methods.

Contribution

It provides the first sharp lower bounds on the complexity of finding stationary points in high-dimensional non-convex optimization, confirming the optimality of existing algorithms.

Findings

Lower bounds match the complexity of gradient descent and Newton's method.

Optimality of first- and higher-order regularization methods is established.

Results apply to a broad class of smooth, non-convex functions.

Abstract

We prove lower bounds on the complexity of finding $\epsilon$-stationary points (points $x$ such that $\|\nabla f(x)\| \le \epsilon$) of smooth, high-dimensional, and potentially non-convex functions $f$. We consider oracle-based complexity measures, where an algorithm is given access to the value and all derivatives of $f$ at a query point $x$. We show that for any (potentially randomized) algorithm $\mathsf{A}$, there exists a function $f$ with Lipschitz $p$th order derivatives such that $\mathsf{A}$ requires at least $\epsilon^{-(p+1)/p}$ queries to find an $\epsilon$-stationary point. Our lower bounds are sharp to within constants, and they show that gradient descent, cubic-regularized Newton's method, and generalized $p$th order regularization are worst-case optimal within their natural function classes.