A Primal-Dual Approach to Bilevel Optimization with Multiple Inner Minima

TL;DR

This paper introduces a primal-dual bilevel optimization algorithm that effectively handles multiple inner minima, providing the first non-asymptotic convergence guarantee and demonstrating strong empirical performance in complex machine learning tasks.

Contribution

It proposes a novel primal-dual approach for bilevel problems with multiple inner minima, offering first-order efficiency and convergence guarantees.

Findings

PDBO effectively solves bilevel problems with multiple inner minima.

The algorithm achieves a non-asymptotic convergence rate.

Experimental results show strong performance in machine learning applications.

Abstract

Bilevel optimization has found extensive applications in modern machine learning problems such as hyperparameter optimization, neural architecture search, meta-learning, etc. While bilevel problems with a unique inner minimal point (e.g., where the inner function is strongly convex) are well understood, such a problem with multiple inner minimal points remains to be challenging and open. Existing algorithms designed for such a problem were applicable to restricted situations and do not come with a full guarantee of convergence. In this paper, we adopt a reformulation of bilevel optimization to constrained optimization, and solve the problem via a primal-dual bilevel optimization (PDBO) algorithm. PDBO not only addresses the multiple inner minima challenge, but also features fully first-order efficiency without involving second-order Hessian and Jacobian computations, as opposed to most existing gradient-based bilevel algorithms. We further characterize the convergence rate of PDBO, which serves as the first known non-asymptotic convergence guarantee for bilevel optimization with multiple inner minima. Our experiments demonstrate desired performance of the proposed approach.