Benchmarking Automatic Machine Learning Frameworks

TL;DR

This paper provides an objective benchmark of popular open-source AutoML frameworks, comparing their performance on classification and regression datasets to guide users in selecting the most effective tools.

Contribution

It introduces a comprehensive benchmark comparing open-source AutoML tools on diverse datasets, highlighting their relative strengths.

Findings

auto-sklearn performs best on classification datasets

TPOT performs best on regression datasets

Provides an objective comparison of AutoML frameworks

Abstract

AutoML serves as the bridge between varying levels of expertise when designing machine learning systems and expedites the data science process. A wide range of techniques is taken to address this, however there does not exist an objective comparison of these techniques. We present a benchmark of current open source AutoML solutions using open source datasets. We test auto-sklearn, TPOT, auto_ml, and H2O's AutoML solution against a compiled set of regression and classification datasets sourced from OpenML and find that auto-sklearn performs the best across classification datasets and TPOT performs the best across regression datasets.