An Iterative Regularized Incremental Projected Subgradient Method for a Class of Bilevel Optimization Problems

TL;DR

This paper introduces a novel iterative regularized incremental subgradient method for solving a new class of bilevel convex optimization problems with a finite sum in the lower level, relevant to distributed machine learning applications.

Contribution

It develops and analyzes the first algorithm specifically designed for bilevel problems with a finite sum in the lower level, providing convergence guarantees and rate analysis.

Findings

Convergence of the proposed algorithm is established under suitable regularization.

The method achieves a rate of O(1/k^{0.5-ε}) in the lower level objective.

Performance demonstrated on a binary text classification task.

Abstract

We study a class of bilevel convex optimization problems where the goal is to find the minimizer of an objective function in the upper level, among the set of all optimal solutions of an optimization problem in the lower level. A wide range of problems in convex optimization can be formulated using this class. An important example is the case where an optimization problem is ill-posed. In this paper, our interest lies in addressing the bilevel problems, where the lower level objective is given as a finite sum of separate nondifferentiable convex component functions. This is the case in a variety of applications in distributed optimization, such as large-scale data processing in machine learning and neural networks. To the best of our knowledge, this class of bilevel problems, with a finite sum in the lower level, has not been addressed before. Motivated by this gap, we develop an iterative regularized incremental subgradient method, where the agents update their iterates in a cyclic manner using a regularized subgradient. Under a suitable choice of the regularization parameter sequence, we establish the convergence of the proposed algorithm and derive a rate of $\mathcal{O} \left({1}/k^{0.5-\epsilon}\right)$ in terms of the lower level objective function for an arbitrary small $\epsilon>0$. We present the performance of the algorithm on a binary text classification problem.