Accelerating Inexact HyperGradient Descent for Bilevel Optimization

TL;DR

This paper introduces RAHGD, a novel accelerated method for nonconvex-strongly-convex bilevel optimization, achieving state-of-the-art theoretical guarantees for finding stationary points with improved oracle complexity.

Contribution

The paper proposes RAHGD, an accelerated hypergradient descent algorithm with optimal complexity bounds for bilevel problems, and extends it to find second-order stationary points.

Findings

Achieves the best-known complexity for stationary point convergence in bilevel optimization.

Improves upper complexity bounds for nonconvex-strongly-concave minimax problems.

Empirical results validate the theoretical improvements.

Abstract

We present a method for solving general nonconvex-strongly-convex bilevel optimization problems. Our method -- the \emph{Restarted Accelerated HyperGradient Descent} (\texttt{RAHGD}) method -- finds an $\epsilon$-first-order stationary point of the objective with $\tilde{\mathcal{O}}(\kappa^{3.25}\epsilon^{-1.75})$ oracle complexity, where $\kappa$ is the condition number of the lower-level objective and $\epsilon$ is the desired accuracy. We also propose a perturbed variant of \texttt{RAHGD} for finding an $\big(\epsilon,\mathcal{O}(\kappa^{2.5}\sqrt{\epsilon}\,)\big)$-second-order stationary point within the same order of oracle complexity. Our results achieve the best-known theoretical guarantees for finding stationary points in bilevel optimization and also improve upon the existing upper complexity bound for finding second-order stationary points in nonconvex-strongly-concave minimax optimization problems, setting a new state-of-the-art benchmark. Empirical studies are conducted to validate the theoretical results in this paper.