Deep Ranking Ensembles for Hyperparameter Optimization

TL;DR

This paper introduces a novel hyperparameter optimization method that uses ranking-based surrogate models trained via ensemble learning, achieving state-of-the-art results across diverse datasets and search spaces.

Contribution

It proposes a new ranking-based approach for training surrogate models in hyperparameter optimization, leveraging ensemble learning to improve performance.

Findings

Achieves state-of-the-art results in HPO across multiple benchmarks.

Outperforms 12 baseline methods in large-scale experiments.

Effectively models uncertainty via ensembling for better ranking accuracy.

Abstract

Automatically optimizing the hyperparameters of Machine Learning algorithms is one of the primary open questions in AI. Existing work in Hyperparameter Optimization (HPO) trains surrogate models for approximating the response surface of hyperparameters as a regression task. In contrast, we hypothesize that the optimal strategy for training surrogates is to preserve the ranks of the performances of hyperparameter configurations as a Learning to Rank problem. As a result, we present a novel method that meta-learns neural network surrogates optimized for ranking the configurations' performances while modeling their uncertainty via ensembling. In a large-scale experimental protocol comprising 12 baselines, 16 HPO search spaces and 86 datasets/tasks, we demonstrate that our method achieves new state-of-the-art results in HPO.