An Open Source AutoML Benchmark

TL;DR

This paper presents an open-source, extensible benchmark framework for AutoML systems, enabling accurate and standardized comparisons across multiple datasets, and demonstrates its use by evaluating four AutoML tools.

Contribution

It introduces a comprehensive, open-source AutoML benchmarking framework that follows best practices and facilitates fair comparisons.

Findings

Compared 4 AutoML systems across 39 datasets

Identified strengths and weaknesses of each system

Provided up-to-date results via a dedicated website

Abstract

In recent years, an active field of research has developed around automated machine learning (AutoML). Unfortunately, comparing different AutoML systems is hard and often done incorrectly. We introduce an open, ongoing, and extensible benchmark framework which follows best practices and avoids common mistakes. The framework is open-source, uses public datasets and has a website with up-to-date results. We use the framework to conduct a thorough comparison of 4 AutoML systems across 39 datasets and analyze the results.