Optimistic Bilevel Optimization with Composite Lower-Level Problem

TL;DR

This paper proposes a new regularization approach for bilevel optimization with composite, convex lower-level problems, providing theoretical guarantees and a gradient sampling algorithm with practical machine learning applications.

Contribution

Introduces a double regularization scheme and gradient sampling algorithm for bilevel problems with composite convex lower-level problems, with convergence analysis and practical demonstrations.

Findings

Gradient of the regularized hyper-objective can be computed almost everywhere.

The proposed method converges to stationary points of the original problem.

Numerical examples validate the effectiveness of the approach.

Abstract

This paper introduces a novel double regularization scheme for bilevel optimization problems whose lower-level problem is composite and convex, but not necessarily strongly convex, in the lower-level variable. The analysis focuses on the primal-dual solution mapping of the regularized lower-level problem and exploits its properties to derive an almost-everywhere formula for the gradient of the regularized hyper-objective under mild assumptions. The paper then establishes conditions under which the hyper-objective of the actual problem is well defined and shows that its gradient can be approximated by the gradient of the regularized hyper-objective. Building on these results, a gradient sampling-based algorithm computes approximately stationary points of the regularized hyper-objective, and we prove its convergence to stationary points of the actual problem. Two numerical examples from machine learning demonstrate the proposed approach.