A General Descent Aggregation Framework for Gradient-based Bi-level Optimization

TL;DR

This paper introduces a unified framework called Bi-level Descent Aggregation (BDA) for gradient-based bi-level optimization, addressing theoretical and practical limitations of existing methods and demonstrating its effectiveness in machine learning tasks.

Contribution

The paper proposes a general, modular algorithmic framework for gradient-based BLO, along with a convergence analysis template applicable to various solution qualities.

Findings

BDA outperforms existing methods in hyper-parameter optimization.

Theoretical convergence guarantees are established for different solution qualities.

Experimental results validate the effectiveness of BDA in practical applications.

Abstract

In recent years, a variety of gradient-based methods have been developed to solve Bi-Level Optimization (BLO) problems in machine learning and computer vision areas. However, the theoretical correctness and practical effectiveness of these existing approaches always rely on some restrictive conditions (e.g., Lower-Level Singleton, LLS), which could hardly be satisfied in real-world applications. Moreover, previous literature only proves theoretical results based on their specific iteration strategies, thus lack a general recipe to uniformly analyze the convergence behaviors of different gradient-based BLOs. In this work, we formulate BLOs from an optimistic bi-level viewpoint and establish a new gradient-based algorithmic framework, named Bi-level Descent Aggregation (BDA), to partially address the above issues. Specifically, BDA provides a modularized structure to hierarchically aggregate both the upper- and lower-level subproblems to generate our bi-level iterative dynamics. Theoretically, we establish a general convergence analysis template and derive a new proof recipe to investigate the essential theoretical properties of gradient-based BLO methods. Furthermore, this work systematically explores the convergence behavior of BDA in different optimization scenarios, i.e., considering various solution qualities (i.e., global/local/stationary solution) returned from solving approximation subproblems. Extensive experiments justify our theoretical results and demonstrate the superiority of the proposed algorithm for hyper-parameter optimization and meta-learning tasks. Source code is available at https://github.com/vis-opt-group/BDA.