Meta-Learning for Symbolic Hyperparameter Defaults

TL;DR

This paper introduces a zero-shot meta-learning approach to automatically derive symbolic hyperparameter defaults based on dataset properties, enabling faster and data-dependent ML configuration without extensive tuning.

Contribution

It presents a novel evolutionary algorithm-based method to learn symbolic hyperparameter defaults from prior evaluations, replacing hand-crafted heuristics.

Findings

Successfully finds viable symbolic defaults across multiple datasets.

Reduces time for hyperparameter configuration compared to traditional methods.

Effective on 6 ML algorithms with over 100 datasets.

Abstract

Hyperparameter optimization in machine learning (ML) deals with the problem of empirically learning an optimal algorithm configuration from data, usually formulated as a black-box optimization problem. In this work, we propose a zero-shot method to meta-learn symbolic default hyperparameter configurations that are expressed in terms of the properties of the dataset. This enables a much faster, but still data-dependent, configuration of the ML algorithm, compared to standard hyperparameter optimization approaches. In the past, symbolic and static default values have usually been obtained as hand-crafted heuristics. We propose an approach of learning such symbolic configurations as formulas of dataset properties from a large set of prior evaluations on multiple datasets by optimizing over a grammar of expressions using an evolutionary algorithm. We evaluate our method on surrogate empirical performance models as well as on real data across 6 ML algorithms on more than 100 datasets and demonstrate that our method indeed finds viable symbolic defaults.