Adaptive Algorithms for Nonconvex Bilevel Optimization under P{\L} Conditions

TL;DR

This paper introduces fully adaptive algorithms for nonconvex bilevel optimization under Polyak-Łojasiewicz conditions, removing the need for problem-specific parameters and achieving optimal iteration and oracle complexities.

Contribution

The paper presents the first fully adaptive methods for nonconvex bilevel optimization under P{ extL} conditions, eliminating the need for prior parameter knowledge.

Findings

Achieve $ ilde{O}(1/ ext{epsilon}^2)$ iteration complexity.

Attain near-optimal first-order oracle complexity.

Match the complexity of gradient descent for single-level problems.

Abstract

Existing methods for nonconvex bilevel optimization (NBO) require prior knowledge of first- and second-order problem-specific parameters (e.g., Lipschitz constants and the Polyak-{\L}ojasiewicz (P{\L}) parameters) to set step sizes, a requirement that poses practical limitations when such parameters are unknown or computationally expensive. We introduce the Adaptive Fully First-order Bilevel Approximation (AF${}^2$BA) algorithm and its accelerated variant, A${}^2$F${}^2$BA, for solving NBO problems under the P{\L} conditions. To our knowledge, these are the first methods to employ fully adaptive step size strategies, eliminating the need for any problem-specific parameters in NBO. We prove that both algorithms achieve $\mathcal{O}(1/\epsilon^2)$ iteration complexity for finding an $\epsilon$-stationary point, matching the iteration complexity of existing well-tuned methods. Furthermore, we show that A${}^2$F${}^2$BA enjoys a near-optimal first-order oracle complexity of $\tilde{\mathcal{O}}(1/\epsilon^2)$, matching the oracle complexity of existing well-tuned methods, and aligning with the complexity of gradient descent for smooth nonconvex single-level optimization when ignoring the logarithmic factors.