First-Order Methods for Linearly Constrained Bilevel Optimization

TL;DR

This paper introduces first-order algorithms for linearly constrained bilevel optimization that achieve near-optimal convergence rates, addressing the computational challenges of Hessian calculations in high-dimensional problems.

Contribution

The authors develop new first-order methods with finite-time guarantees for constrained bilevel problems, including nearly-optimal and dimension-free convergence rates.

Findings

Achieve $ ilde{O}(rac{1}{\e^2})$ gradient calls for linear equality constraints.

Attain $(\delta, ext{ extepsilon})$-Goldstein stationarity in $ ilde{O}(d ext{ extdelta}^{-1} ext{ extepsilon}^{-3})$ calls for linear inequality constraints.

Dimension-free rates of $ ilde{O}( ext{ extdelta}^{-1} ext{ extepsilon}^{-4})$ oracle complexity under additional dual variable access.

Abstract

Algorithms for bilevel optimization often encounter Hessian computations, which are prohibitive in high dimensions. While recent works offer first-order methods for unconstrained bilevel problems, the constrained setting remains relatively underexplored. We present first-order linearly constrained optimization methods with finite-time hypergradient stationarity guarantees. For linear equality constraints, we attain $\epsilon$-stationarity in $\widetilde{O}(\epsilon^{-2})$ gradient oracle calls, which is nearly-optimal. For linear inequality constraints, we attain $(\delta,\epsilon)$-Goldstein stationarity in $\widetilde{O}(d{\delta^{-1} \epsilon^{-3}})$ gradient oracle calls, where $d$ is the upper-level dimension. Finally, we obtain for the linear inequality setting dimension-free rates of $\widetilde{O}({\delta^{-1} \epsilon^{-4}})$ oracle complexity under the additional assumption of oracle access to the optimal dual variable. Along the way, we develop new nonsmooth nonconvex optimization methods with inexact oracles. We verify these guarantees with preliminary numerical experiments.