On the Lower Bound of Minimizing Polyak-{\L}ojasiewicz Functions

TL;DR

This paper establishes a lower bound on the gradient complexity for first-order algorithms minimizing Polyak-Łojasiewicz functions, showing that Gradient Descent is optimal in this setting and highlighting the hardness of acceleration techniques.

Contribution

It proves a fundamental lower bound on the gradient complexity for first-order methods on PL functions, demonstrating the optimality of Gradient Descent and distinguishing it from strongly convex functions.

Findings

Gradient Descent is optimal for minimizing smooth PL functions.

Any first-order algorithm requires at least Ω(L/μ log(1/ε)) gradient evaluations.

Acceleration techniques cannot improve the complexity beyond this lower bound for PL functions.

Abstract

Polyak-{\L}ojasiewicz (PL) [Polyak, 1963] condition is a weaker condition than the strong convexity but suffices to ensure a global convergence for the Gradient Descent algorithm. In this paper, we study the lower bound of algorithms using first-order oracles to find an approximate optimal solution. We show that any first-order algorithm requires at least ${\Omega}\left(\frac{L}{\mu}\log\frac{1}{\varepsilon}\right)$ gradient costs to find an $\varepsilon$-approximate optimal solution for a general $L$-smooth function that has an $\mu$-PL constant. This result demonstrates the optimality of the Gradient Descent algorithm to minimize smooth PL functions in the sense that there exists a ``hard'' PL function such that no first-order algorithm can be faster than Gradient Descent when ignoring a numerical constant. In contrast, it is well-known that the momentum technique, e.g. [Nesterov, 2003, chap. 2] can provably accelerate Gradient Descent to ${O}\left(\sqrt{\frac{L}{\hat{\mu}}}\log\frac{1}{\varepsilon}\right)$ gradient costs for functions that are $L$-smooth and $\hat{\mu}$-strongly convex. Therefore, our result distinguishes the hardness of minimizing a smooth PL function and a smooth strongly convex function as the complexity of the former cannot be improved by any polynomial order in general.