Unlocking Global Optimality in Bilevel Optimization: A Pilot Study

TL;DR

This paper investigates the challenge of achieving global optimality in bilevel optimization, proposing conditions for convergence and validating them through experiments in machine learning scenarios.

Contribution

It introduces two sufficient conditions for global convergence in bilevel optimization and demonstrates their effectiveness in specific learning applications.

Findings

Proposed two sufficient conditions for global convergence.

Validated conditions through experiments in representation learning and hypercleaning.

Confirmed convergence to global minima in tested scenarios.

Abstract

Bilevel optimization has witnessed a resurgence of interest, driven by its critical role in trustworthy and efficient AI applications. While many recent works have established convergence to stationary points or local minima, obtaining the global optimum of bilevel optimization remains an important yet open problem. The difficulty lies in the fact that, unlike many prior non-convex single-level problems, bilevel problems often do not admit a benign landscape, and may indeed have multiple spurious local solutions. Nevertheless, attaining global optimality is indispensable for ensuring reliability, safety, and cost-effectiveness, particularly in high-stakes engineering applications that rely on bilevel optimization. In this paper, we first explore the challenges of establishing a global convergence theory for bilevel optimization, and present two sufficient conditions for global convergence. We provide algorithm-dependent proofs to rigorously substantiate these sufficient conditions on two specific bilevel learning scenarios: representation learning and data hypercleaning (a.k.a. reweighting). Experiments corroborate the theoretical findings, demonstrating convergence to the global minimum in both cases.