On the Convergence Rates of Iterative Regularization Algorithms for Composite Bi-Level Optimization

TL;DR

This paper analyzes the convergence rates of iterative regularization algorithms for composite bi-level optimization, introducing novel theoretical results, accelerated variants, and surrogate schemes to handle computational challenges.

Contribution

It provides the first simultaneous convergence rate analysis for the IRE-PG method and extends it to an accelerated version with practical surrogate schemes.

Findings

Accelerated IRE-PG achieves faster convergence under certain conditions.

Surrogate schemes enable handling of complex bi-level problems with approximate proximal operators.

Convergence rates for inner and outer functions reveal inherent trade-offs.

Abstract

This paper investigates iterative methods for solving bi-level optimization problems where both inner and outer functions have a composite structure. We establish novel theoretical results, including the first analysis that provides simultaneous convergence rates for the Iteratively REgularized Proximal Gradient (IRE-PG) method, a variant of Solodov's algorithm. These rates for the inner and outer functions highlight the inherent trade-offs between their respective convergence behaviors. We further extend this analysis to an accelerated version of IRE-PG, proving faster convergence rates under specific settings. Additionally, we propose a new scheme for handling cases where these methods cannot be directly applied to the bi-level problem due to the difficulty of computing the associated proximal operator. This scheme offers surrogate functions to approximate the original problem and a framework to translate convergence rates between the surrogate and original functions. Our results show that the accelerated method's advantage diminishes under this translation.