Second-Order Bilevel Optimization with Accelerated Convergence Rates

TL;DR

This paper introduces a second-order bilevel optimization method with accelerated convergence rates and a lazy variant that reduces computational costs, demonstrating improved efficiency over first-order methods.

Contribution

The paper proposes a novel fully second-order bilevel approximation method (FSBA) with accelerated convergence and a lazy variant (LFSBA) that reduces computational complexity, also extending ideas to minimax optimization.

Findings

FSBA achieves an iteration complexity of approximately $ ilde{O}(rac{1}{ ext{epsilon}^{1.5}})$.

LFSBA reuses second-order information, reducing complexity by a factor of $ extsqrt{d}$.

The methods outperform existing first- and second-order approaches in convergence and computational efficiency.

Abstract

This paper studies second-order methods for nonconvex-strongly-convex bilevel optimization. We propose a novel fully second-order bilevel approximation method (FSBA) that achieves an iteration complexity of $\tilde{\mathcal{O}}(\epsilon^{-1.5})$ for finding the $(\epsilon, \mathcal{O}(\sqrt{\epsilon}))$ second-order stationary point of the hyper-objective function. Our results demonstrate that second-order methods can achieve an accelerated convergence rate than first-order methods in bilevel optimization. To address the heavy computational cost associated with the second-order oracle, we introduce a lazy variant of FSBA, called LFSBA, which reuses second-order information across several iterations. We prove that LFSBA exhibits better computational complexity than FSBA by a factor of $\sqrt{d}$, where $d$ is the dimension of the problem. We also apply a similar idea to nonconvex strongly-concave minimax optimization and propose the lazy minimax cubic-regularized Newton (LMCN) method with better computational complexity compared to existing second-order methods.