Stabilizing Bi-Level Hyperparameter Optimization using Moreau-Yosida Regularization

TL;DR

This paper introduces the MY-HPO algorithm that employs Moreau-Yosida regularization to stabilize and improve convergence in bi-level hyperparameter optimization, demonstrating significant empirical performance gains.

Contribution

The paper presents a novel regularization-based algorithm for bi-level HPO, with theoretical analysis and empirical validation showing improved stability and results.

Findings

Significant reduction in loss values compared to existing solvers

Theoretical proof of solution correctness and initial convergence guarantees

Enhanced stability in hyperparameter optimization processes

Abstract

This research proposes to use the Moreau-Yosida envelope to stabilize the convergence behavior of bi-level Hyperparameter optimization solvers, and introduces the new algorithm called Moreau-Yosida regularized Hyperparameter Optimization (MY-HPO) algorithm. Theoretical analysis on the correctness of the MY-HPO solution and initial convergence analysis is also provided. Our empirical results show significant improvement in loss values for a fixed computation budget, compared to the state-of-art bi-level HPO solvers.