On the Condition Number Dependency in Bilevel Optimization

TL;DR

This paper investigates the complexity bounds of bilevel optimization with nonconvex upper-level and strongly convex lower-level problems, establishing new lower and upper bounds that reveal a gap from minimax problems and extend to various settings.

Contribution

It introduces the first provable gap between bilevel and minimax problems in condition number dependency and extends bounds to high-order, stochastic, and convex hyper-objective scenarios.

Findings

Established a $ ilde{ ext{O}}( ext{condition number}^{7/2} imes ext{epsilon}^{-2})$ upper bound.

Proved a $ ext{Omega}( ext{condition number}^2 imes ext{epsilon}^{-2})$ lower bound.

Extended bounds to high-order smooth, stochastic, and convex hyper-objective problems.

Abstract

Bilevel optimization minimizes an objective function, defined by an upper-level problem whose feasible region is the solution of a lower-level problem. We study the oracle complexity of finding an $\epsilon$-stationary point with first-order methods when the upper-level problem is nonconvex and the lower-level problem is strongly convex. Recent works (Ji et al., ICML 2021; Arbel and Mairal, ICLR 2022; Chen el al., JMLR 2025) achieve a $\tilde{\mathcal{O}}(\kappa^4 \epsilon^{-2})$ upper bound that is near-optimal in $\epsilon$. However, the optimal dependency on the condition number $\kappa$ is unknown. In this work, we establish a new $\Omega(\kappa^2 \epsilon^{-2})$ lower bound and $\tilde{\mathcal{O}}(\kappa^{7/2} \epsilon^{-2})$ upper bound for this problem, establishing the first provable gap between bilevel problems and minimax problems in this setup. Our lower bounds can be extended to various settings, including high-order smooth functions, stochastic oracles, and convex hyper-objectives: (1) For second-order and arbitrarily smooth problems, we show $\Omega(\kappa_y^{13/4} \epsilon^{-12/7})$ and $\Omega(\kappa^{17/10} \epsilon^{-8/5})$ lower bounds, respectively. (2) For convex-strongly-convex problems, we improve the previously best lower bound (Ji and Liang, JMLR 2022) from $\Omega(\kappa /\sqrt{\epsilon})$ to $\Omega(\kappa^{5/4} / \sqrt{\epsilon})$. (3) For smooth stochastic problems, we show an $\Omega(\kappa^4 \epsilon^{-4})$ lower bound.