Bilevel Optimization with Lower-Level Uniform Convexity: Theory and Algorithm

TL;DR

This paper introduces a new class of bilevel optimization problems with lower-level uniform convexity, providing a novel theoretical framework and an efficient stochastic algorithm with provable convergence guarantees.

Contribution

It characterizes the smoothness of the hyperobjective under lower-level uniform convexity and proposes UniBiO, a stochastic algorithm with optimal complexity bounds for this class.

Findings

Established a new implicit differentiation theorem for uniformly convex lower-level functions.

Designed UniBiO, a stochastic algorithm with convergence guarantees.

Achieved near-optimal oracle complexity bounds matching known rates.

Abstract

Bilevel optimization is a hierarchical framework where an upper-level optimization problem is constrained by a lower-level problem, commonly used in machine learning applications such as hyperparameter optimization. Existing bilevel optimization methods typically assume strong convexity or Polyak-{\L}ojasiewicz (PL) conditions for the lower-level function to establish non-asymptotic convergence to a solution with small hypergradient. However, these assumptions may not hold in practice, and recent work~\citep{chen2024finding} has shown that bilevel optimization is inherently intractable for general convex lower-level functions with the goal of finding small hypergradients. In this paper, we identify a tractable class of bilevel optimization problems that interpolates between lower-level strong convexity and general convexity via \emph{lower-level uniform convexity}. For uniformly convex lower-level functions with exponent $p\geq 2$, we establish a novel implicit differentiation theorem characterizing the hyperobjective's smoothness property. Building on this, we design a new stochastic algorithm, termed UniBiO, with provable convergence guarantees, based on an oracle that provides stochastic gradient and Hessian-vector product information for the bilevel problems. Our algorithm achieves $\widetilde{O}(\epsilon^{-5p+6})$ oracle complexity bound for finding $\epsilon$-stationary points. Notably, our complexity bounds match the optimal rates in terms of the $\epsilon$ dependency for strongly convex lower-level functions ($p=2$), up to logarithmic factors. Our theoretical findings are validated through experiments on synthetic tasks and data hyper-cleaning, demonstrating the effectiveness of our proposed algorithm.