Alternating Gradient-Type Algorithm for Bilevel Optimization with Inexact Lower-Level Solutions via Moreau Envelope-based Reformulation

TL;DR

This paper introduces AGILS, an efficient bilevel optimization algorithm that handles inexact lower-level solutions using a Moreau envelope reformulation, with proven convergence and practical effectiveness.

Contribution

Proposes a novel AGILS algorithm for bilevel optimization with inexact solutions, leveraging Moreau envelope reformulation and establishing convergence properties.

Findings

Proves convergence of AGILS to stationary points.

Demonstrates effectiveness through numerical experiments.

Handles inexact lower-level solutions efficiently.

Abstract

In this paper, we study a class of bilevel optimization problems where the lower-level problem is a convex composite optimization model, which arises in various applications, including bilevel hyperparameter selection for regularized regression models. To solve these problems, we propose an Alternating Gradient-type algorithm with Inexact Lower-level Solutions (AGILS) based on a Moreau envelope-based reformulation of the bilevel optimization problem. The proposed algorithm does not require exact solutions of the lower-level problem at each iteration, improving computational efficiency. We prove the convergence of AGILS to stationary points and, under the Kurdyka-{\L}ojasiewicz (KL) property, establish its sequential convergence. Numerical experiments, including a toy example and a bilevel hyperparameter selection problem for the sparse group Lasso model, demonstrate the effectiveness of the proposed AGILS.