Lower Complexity Bounds for Nonconvex-Strongly-Convex Bilevel Optimization with First-Order Oracles

TL;DR

This paper establishes new lower bounds on the number of first-order oracle calls needed for nonconvex-strongly-convex bilevel optimization, revealing gaps between existing algorithms and theoretical limits in both deterministic and stochastic settings.

Contribution

It introduces novel hard instances to derive lower bounds, improving understanding of the fundamental complexity of bilevel optimization with first-order oracles.

Findings

Deterministic case lower bound: Ω(κ^{3/2} ε^{-2}) oracle calls.

Stochastic case lower bound: Ω(κ^{5/2} ε^{-4}) oracle calls.

Highlights gaps between current algorithms and theoretical lower bounds.

Abstract

Although upper bound guarantees for bilevel optimization have been widely studied, progress on lower bounds has been limited due to the complexity of the bilevel structure. In this work, we focus on the smooth nonconvex-strongly-convex setting and develop new hard instances that yield nontrivial lower bounds under deterministic and stochastic first-order oracle models. In the deterministic case, we prove that any first-order zero-respecting algorithm requires at least $\Omega(\kappa^{3/2}\epsilon^{-2})$ oracle calls to find an $\epsilon$-accurate stationary point, improving the optimal lower bounds known for single-level nonconvex optimization and for nonconvex-strongly-convex min-max problems. In the stochastic case, we show that at least $\Omega(\kappa^{5/2}\epsilon^{-4})$ stochastic oracle calls are necessary, again strengthening the best known bounds in related settings. Our results expose substantial gaps between current upper and lower bounds for bilevel optimization and suggest that even simplified regimes, such as those with quadratic lower-level objectives, warrant further investigation toward understanding the optimal complexity of bilevel optimization under standard first-order oracles.