Mining Robust Default Configurations for Resource-constrained AutoML

TL;DR

This paper introduces a zero-shot AutoML configuration selection method that mines diverse tasks to identify robust default configurations, improving performance across multiple datasets without online training.

Contribution

The paper presents a novel offline mining approach for selecting default AutoML configurations that generalize well to unseen tasks, enhancing efficiency and effectiveness.

Findings

Outperforms state-of-the-art on 62 datasets

Effective for warm-starting existing AutoML platforms

Recommends data-dependent default configurations

Abstract

Automatic machine learning (AutoML) is a key enabler of the mass deployment of the next generation of machine learning systems. A key desideratum for future ML systems is the automatic selection of models and hyperparameters. We present a novel method of selecting performant configurations for a given task by performing offline autoML and mining over a diverse set of tasks. By mining the training tasks, we can select a compact portfolio of configurations that perform well over a wide variety of tasks, as well as learn a strategy to select portfolio configurations for yet-unseen tasks. The algorithm runs in a zero-shot manner, that is without training any models online except the chosen one. In a compute- or time-constrained setting, this virtually instant selection is highly performant. Further, we show that our approach is effective for warm-starting existing autoML platforms. In both settings, we demonstrate an improvement on the state-of-the-art by testing over 62 classification and regression datasets. We also demonstrate the utility of recommending data-dependent default configurations that outperform widely used hand-crafted defaults.