Moreau Envelope Based Difference-of-weakly-Convex Reformulation and Algorithm for Bilevel Programs

TL;DR

This paper introduces a novel reformulation and algorithm for bilevel programs that only require convexity in the lower-level variables, broadening applicability in machine learning hyperparameter tuning.

Contribution

It presents a new single-level difference of weakly convex reformulation using the Moreau envelope and develops a convergent inexact proximal algorithm for bilevel optimization.

Findings

Effective hyperparameter tuning for SVMs demonstrated

Broader application scope beyond fully convex lower-level problems

Convergence of the proposed algorithm confirmed

Abstract

Bilevel programming has emerged as a valuable tool for hyperparameter selection, a central concern in machine learning. In a recent study by Ye et al. (2023), a value function-based difference of convex algorithm was introduced to address bilevel programs. This approach proves particularly powerful when dealing with scenarios where the lower-level problem exhibits convexity in both the upper-level and lower-level variables. Examples of such scenarios include support vector machines and $\ell_1$ and $\ell_2$ regularized regression. In this paper, we significantly expand the range of applications, now requiring convexity only in the lower-level variables of the lower-level program. We present an innovative single-level difference of weakly convex reformulation based on the Moreau envelope of the lower-level problem. We further develop a sequentially convergent Inexact Proximal Difference of Weakly Convex Algorithm (iP-DwCA). To evaluate the effectiveness of the proposed iP-DwCA, we conduct numerical experiments focused on tuning hyperparameters for kernel support vector machines on simulated data.