Automated Benchmark-Driven Design and Explanation of Hyperparameter Optimizers

TL;DR

This paper introduces a systematic, benchmark-driven approach to designing hyperparameter optimizers using Bayesian optimization, formalizing a rich candidate space and demonstrating that simpler configurations can perform effectively.

Contribution

It presents a principled, automated framework for hyperparameter optimizer design that reduces reliance on manual, arbitrary choices and enables systematic exploration of algorithm components.

Findings

Simple configurations can perform competitively with complex methods.

Benchmark-driven design helps identify critical parameters.

Ablation analysis clarifies the necessity of certain design choices.

Abstract

Automated hyperparameter optimization (HPO) has gained great popularity and is an important ingredient of most automated machine learning frameworks. The process of designing HPO algorithms, however, is still an unsystematic and manual process: Limitations of prior work are identified and the improvements proposed are -- even though guided by expert knowledge -- still somewhat arbitrary. This rarely allows for gaining a holistic understanding of which algorithmic components are driving performance, and carries the risk of overlooking good algorithmic design choices. We present a principled approach to automated benchmark-driven algorithm design applied to multifidelity HPO (MF-HPO): First, we formalize a rich space of MF-HPO candidates that includes, but is not limited to common HPO algorithms, and then present a configurable framework covering this space. To find the best candidate automatically and systematically, we follow a programming-by-optimization approach and search over the space of algorithm candidates via Bayesian optimization. We challenge whether the found design choices are necessary or could be replaced by more naive and simpler ones by performing an ablation analysis. We observe that using a relatively simple configuration, in some ways simpler than established methods, performs very well as long as some critical configuration parameters have the right value.