Towards Extremely Fast Bilevel Optimization with Self-governed Convergence Guarantees

TL;DR

This paper introduces BAGDC, a novel framework for bilevel optimization that accelerates existing gradient-based methods and guarantees convergence, significantly reducing computational costs in learning and vision tasks.

Contribution

The paper proposes a unified single-level formulation and the BAGDC framework, providing convergence guarantees and accelerating existing gradient-based bilevel optimization algorithms.

Findings

BAGDC significantly speeds up bilevel optimization algorithms.

Theoretical convergence guarantees are established for BAGDC.

Numerical experiments demonstrate the effectiveness of the proposed method.

Abstract

Gradient methods have become mainstream techniques for Bi-Level Optimization (BLO) in learning and vision fields. The validity of existing works heavily relies on solving a series of approximation subproblems with extraordinarily high accuracy. Unfortunately, to achieve the approximation accuracy requires executing a large quantity of time-consuming iterations and computational burden is naturally caused. This paper is thus devoted to address this critical computational issue. In particular, we propose a single-level formulation to uniformly understand existing explicit and implicit Gradient-based BLOs (GBLOs). This together with our designed counter-example can clearly illustrate the fundamental numerical and theoretical issues of GBLOs and their naive accelerations. By introducing the dual multipliers as a new variable, we then establish Bilevel Alternating Gradient with Dual Correction (BAGDC), a general framework, which significantly accelerates different categories of existing methods by taking specific settings. A striking feature of our convergence result is that, compared to those original unaccelerated GBLO versions, the fast BAGDC admits a unified non-asymptotic convergence theory towards stationarity. A variety of numerical experiments have also been conducted to demonstrate the superiority of the proposed algorithmic framework.