A Gradient Method for Multilevel Optimization

TL;DR

This paper introduces a gradient-based algorithm for multilevel optimization with theoretical convergence guarantees, extending previous bilevel methods and demonstrating improved stability in hyperparameter learning under noisy data conditions.

Contribution

It develops one of the first algorithms with theoretical guarantees for multilevel optimization, generalizing bilevel approaches to multiple levels.

Findings

Algorithm converges asymptotically to the multilevel problem

Outperforms bilevel models in noisy data hyperparameter learning

Provides theoretical foundation for multilevel optimization methods

Abstract

Although application examples of multilevel optimization have already been discussed since the 1990s, the development of solution methods was almost limited to bilevel cases due to the difficulty of the problem. In recent years, in machine learning, Franceschi et al. have proposed a method for solving bilevel optimization problems by replacing their lower-level problems with the $T$ steepest descent update equations with some prechosen iteration number $T$. In this paper, we have developed a gradient-based algorithm for multilevel optimization with $n$ levels based on their idea and proved that our reformulation asymptotically converges to the original multilevel problem. As far as we know, this is one of the first algorithms with some theoretical guarantee for multilevel optimization. Numerical experiments show that a trilevel hyperparameter learning model considering data poisoning produces more stable prediction results than an existing bilevel hyperparameter learning model in noisy data settings.