Solving bilevel optimization via sequential minimax optimization

TL;DR

This paper introduces a sequential minimax optimization method for bilevel problems with nonsmooth convex lower levels and nonconvex upper levels, achieving improved complexity bounds and demonstrating superior numerical performance.

Contribution

The paper proposes a novel SMO algorithm combining augmented Lagrangian and penalty schemes, with improved complexity bounds for solving bilevel optimization problems.

Findings

Achieves operation complexity of $O( ext{epsilon}^{-7} ext{log} ext{epsilon}^{-1})$ for convex lower levels.

Achieves operation complexity of $O( ext{epsilon}^{-6} ext{log} ext{epsilon}^{-1})$ for strongly convex lower levels.

Numerical results show significantly better performance than existing penalty methods.

Abstract

In this paper we propose a sequential minimax optimization (SMO) method for solving a class of constrained bilevel optimization problems in which the lower-level part is a possibly nonsmooth convex optimization problem, while the upper-level part is a possibly nonconvex optimization problem. Specifically, SMO applies a first-order method to solve a sequence of minimax subproblems, which are obtained by employing a hybrid of modified augmented Lagrangian and penalty schemes on the bilevel optimization problems. Under suitable assumptions, we establish an operation complexity of $O(\varepsilon^{-7}\log\varepsilon^{-1})$ and $O(\varepsilon^{-6}\log\varepsilon^{-1})$, measured in terms of fundamental operations, for SMO in finding an $\varepsilon$-KKT solution of the bilevel optimization problems with merely convex and strongly convex lower-level objective functions, respectively. The latter result improves the previous best-known operation complexity by a factor of $\varepsilon^{-1}$. Preliminary numerical results demonstrate significantly superior computational performance compared to the recently developed first-order penalty method.