Hyperparameter Tuning Through Pessimistic Bilevel Optimization

TL;DR

This paper introduces a pessimistic bilevel hyperparameter optimization method that explicitly accounts for model uncertainty, leading to more robust hyperparameters and improved generalization in limited or perturbed data scenarios.

Contribution

It proposes a novel pessimistic bilevel optimization framework for hyperparameter tuning that considers uncertainty, along with a relaxation-based approximation method to solve it.

Findings

Pessimistic solutions outperform optimistic ones with limited training data.

Pessimistic approach yields more robust prediction models.

Empirical results show improved generalization under data perturbations.

Abstract

Automated hyperparameter search in machine learning, especially for deep learning models, is typically formulated as a bilevel optimization problem, with hyperparameter values determined by the upper level and the model learning achieved by the lower-level problem. Most of the existing bilevel optimization solutions either assume the uniqueness of the optimal training model given hyperparameters or adopt an optimistic view when the non-uniqueness issue emerges. Potential model uncertainty may arise when training complex models with limited data, especially when the uniqueness assumption is violated. Thus, the suitability of the optimistic view underlying current bilevel hyperparameter optimization solutions is questionable. In this paper, we propose pessimistic bilevel hyperparameter optimization to assure appropriate outer-level hyperparameters to better generalize the inner-level learned models, by explicitly incorporating potential uncertainty of the inner-level solution set. To solve the resulting computationally challenging pessimistic bilevel optimization problem, we develop a novel relaxation-based approximation method. It derives pessimistic solutions with more robust prediction models. In our empirical studies of automated hyperparameter search for binary linear classifiers, pessimistic solutions have demonstrated better prediction performances than optimistic counterparts when we have limited training data or perturbed testing data, showing the necessity of considering pessimistic solutions besides existing optimistic ones.