Dynamic Design of Machine Learning Pipelines via Metalearning

TL;DR

This paper presents a metalearning approach to dynamically design AutoML search spaces, significantly reducing runtime and search space size while maintaining predictive performance.

Contribution

It introduces a novel metalearning method that uses historical meta-knowledge to adaptively select promising search regions for AutoML, improving efficiency.

Findings

Reduced AutoML runtime by 89%

Shrank search space by up to 4.3/16 classifiers

Maintained competitive predictive performance

Abstract

Automated machine learning (AutoML) has democratized the design of machine learning based systems, by automating model selection, hyperparameter tuning and feature engineering. However, the high computational cost associated with traditional search and optimization strategies, such as Random Search, Particle Swarm Optimization and Bayesian Optimization, remains a significant challenge. Moreover, AutoML systems typically explore a large search space, which can lead to overfitting. This paper introduces a metalearning method for dynamically designing search spaces for AutoML system. The proposed method uses historical metaknowledge to select promising regions of the search space, accelerating the optimization process. According to experiments conducted for this study, the proposed method can reduce runtime by 89\% in Random Search and search space by (1.8/13 preprocessor and 4.3/16 classifier), without compromising significant predictive performance. Moreover, the proposed method showed competitive performance when adapted to Auto-Sklearn, reducing its search space. Furthermore, this study encompasses insights into meta-feature selection, meta-model explainability, and the trade-offs inherent in search space reduction strategies.