qNBO: quasi-Newton Meets Bilevel Optimization

TL;DR

This paper introduces qNBO, a framework combining quasi-Newton methods with bilevel optimization to improve computational efficiency and convergence in hierarchical learning tasks, demonstrated through various real-world applications.

Contribution

It presents a novel framework that integrates quasi-Newton algorithms into bilevel optimization, enabling faster lower-level problem solving and inverse Hessian approximation with proven convergence.

Findings

Comparable or superior performance in real-world tasks

Efficient approximation of inverse Hessian-vector products

Non-asymptotic convergence analysis of BFGS in this context

Abstract

Bilevel optimization, addressing challenges in hierarchical learning tasks, has gained significant interest in machine learning. The practical implementation of the gradient descent method to bilevel optimization encounters computational hurdles, notably the computation of the exact lower-level solution and the inverse Hessian of the lower-level objective. Although these two aspects are inherently connected, existing methods typically handle them separately by solving the lower-level problem and a linear system for the inverse Hessian-vector product. In this paper, we introduce a general framework to address these computational challenges in a coordinated manner. Specifically, we leverage quasi-Newton algorithms to accelerate the resolution of the lower-level problem while efficiently approximating the inverse Hessian-vector product. Furthermore, by exploiting the superlinear convergence properties of BFGS, we establish the non-asymptotic convergence analysis of the BFGS adaptation within our framework. Numerical experiments demonstrate the comparable or superior performance of the proposed algorithms in real-world learning tasks, including hyperparameter optimization, data hyper-cleaning, and few-shot meta-learning.