AutoMLBench: A Comprehensive Experimental Evaluation of Automated Machine Learning Frameworks

TL;DR

This paper provides a comprehensive experimental comparison of six popular AutoML frameworks across 100 datasets, analyzing their performance and design choices to guide future AutoML development.

Contribution

It offers the first extensive evaluation of multiple AutoML frameworks on a large benchmark, highlighting key factors influencing their performance.

Findings

Performance varies significantly across frameworks and datasets.

Design decisions like search space size and ensemble methods impact results.

Meta-learning and time budget influence AutoML effectiveness.

Abstract

With the booming demand for machine learning applications, it has been recognized that the number of knowledgeable data scientists can not scale with the growing data volumes and application needs in our digital world. In response to this demand, several automated machine learning (AutoML) frameworks have been developed to fill the gap of human expertise by automating the process of building machine learning pipelines. Each framework comes with different heuristics-based design decisions. In this study, we present a comprehensive evaluation and comparison of the performance characteristics of six popular AutoML frameworks, namely, AutoWeka, AutoSKlearn, TPOT, Recipe, ATM, and SmartML, across 100 data sets from established AutoML benchmark suites. Our experimental evaluation considers different aspects for its comparison, including the performance impact of several design decisions, including time budget, size of search space, meta-learning, and ensemble construction. The results of our study reveal various interesting insights that can significantly guide and impact the design of AutoML frameworks.