Are encoders able to learn landmarkers for warm-starting of Hyperparameter Optimization?

TL;DR

This paper investigates whether encoder models can learn effective landmarkers for warm-starting Bayesian Hyperparameter Optimization, proposing two novel representation learning methods and evaluating their alignment and impact.

Contribution

Introduces two new tabular representation learning methods tailored for meta-learning in HPO, focusing on capturing properties of landmarkers.

Findings

Encoders can learn representations aligned with landmarkers.

Alignment does not necessarily lead to significant HPO performance improvements.

Proposed methods effectively capture landmarkers' properties.

Abstract

Effectively representing heterogeneous tabular datasets for meta-learning purposes is still an open problem. Previous approaches rely on representations that are intended to be universal. This paper proposes two novel methods for tabular representation learning tailored to a specific meta-task - warm-starting Bayesian Hyperparameter Optimization. Both follow the specific requirement formulated by ourselves that enforces representations to capture the properties of landmarkers. The first approach involves deep metric learning, while the second one is based on landmarkers reconstruction. We evaluate the proposed encoders in two ways. Next to the gain in the target meta-task, we also use the degree of fulfillment of the proposed requirement as the evaluation metric. Experiments demonstrate that while the proposed encoders can effectively learn representations aligned with landmarkers, they may not directly translate to significant performance gains in the meta-task of HPO warm-starting.