HPO X ELA: Investigating Hyperparameter Optimization Landscapes by Means of Exploratory Landscape Analysis
Lennart Schneider, Lennart Sch\"apermeier, Raphael Patrick Prager,, Bernd Bischl, Heike Trautmann, Pascal Kerschke

TL;DR
This paper applies exploratory landscape analysis to hyperparameter optimization problems, comparing their structure to benchmark problems and evaluating optimizer performance to better understand the landscape of HPO tasks.
Contribution
It introduces a methodology to analyze HPO landscapes using ELA features and compares them with benchmark problems, revealing structural similarities and differences.
Findings
HPO problems share structural features with certain BBOB benchmark problems.
Optimizer performance correlates with landscape similarity in ELA feature space.
Identifies open challenges and future directions for ELA in HPO.
Abstract
Hyperparameter optimization (HPO) is a key component of machine learning models for achieving peak predictive performance. While numerous methods and algorithms for HPO have been proposed over the last years, little progress has been made in illuminating and examining the actual structure of these black-box optimization problems. Exploratory landscape analysis (ELA) subsumes a set of techniques that can be used to gain knowledge about properties of unknown optimization problems. In this paper, we evaluate the performance of five different black-box optimizers on 30 HPO problems, which consist of two-, three- and five-dimensional continuous search spaces of the XGBoost learner trained on 10 different data sets. This is contrasted with the performance of the same optimizers evaluated on 360 problem instances from the black-box optimization benchmark (BBOB). We then compute ELA features on…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Multi-Objective Optimization Algorithms · Machine Learning and Data Classification · Metaheuristic Optimization Algorithms Research
MethodsHyper-parameter optimization
