ADMM-based Bilevel Descent Aggregation Algorithm for Sparse Hyperparameter Selection

TL;DR

This paper introduces an ADMM-based bilevel descent aggregation algorithm that effectively solves sparse hyperparameter selection problems, achieving global convergence without the lower-level singleton assumption and demonstrating superior performance in experiments.

Contribution

It presents a novel ADMM-BDA framework for bilevel optimization that relaxes traditional assumptions and provides convergence guarantees.

Findings

Achieves global convergence under relaxed conditions.

Outperforms state-of-the-art algorithms in experiments.

Effective for elastic-net penalized problems.

Abstract

It is widely acknowledged that hyperparameter selection plays a critical role in the effectiveness of sparse optimization problems. The bilevel optimization provides a robust framework for addressing this issue, but these existing methods depend heavily on the lower-level singleton (LLS) assumption, which greatly limits their practical applicabilities. To tackle this technical challenge, this paper focus on a particular type of nonsmooth convex sparse optimization problem and presents a new bilevel optimization framework. This framework effectively integrates the alternating direction method of multipliers (ADMM) with a bilevel descent aggregation (BDA) algorithm. Specifically, it employs ADMM to efficiently address the lower-level problem and uses BDA to explore the hyperparameter space, thereby integrating both the upper and lower-level problems. It is important to emphasize that a key contribution of this paper lies in the presentation of a novel convergence analysis. The analysis illustrates that the proposed ADMM-BDA algorithm achieves global convergence under significantly relaxed conditions, thereby departing from the LLS assumption that are often required in the literature. We conduct a series of numerical experiments utilizing synthetic and real-world data, and do performance comparisions against some state-of-the-art algorithms. The results indicates that ADMM-BDA exhibits superior effectiveness and robustness for solving bilevel programming problems, especially when the lower-level problem is an elastic-net penalized statistics problem.