OpenML Benchmarking Suites

TL;DR

This paper advocates for standardized, comprehensive benchmarking suites in machine learning, introduces the OpenML platform for creating and sharing these benchmarks, and presents a curated classification suite for practical use.

Contribution

It introduces OpenML benchmarking suites, including a curated classification benchmark, enabling standardized, shareable, and reproducible machine learning evaluations.

Findings

OpenML suites are easy to use with standardized formats and APIs

The curated classification suite facilitates practical benchmarking

OpenML promotes sharing and reuse of benchmarking results

Abstract

Machine learning research depends on objectively interpretable, comparable, and reproducible algorithm benchmarks. We advocate the use of curated, comprehensive suites of machine learning tasks to standardize the setup, execution, and reporting of benchmarks. We enable this through software tools that help to create and leverage these benchmarking suites. These are seamlessly integrated into the OpenML platform, and accessible through interfaces in Python, Java, and R. OpenML benchmarking suites (a) are easy to use through standardized data formats, APIs, and client libraries; (b) come with extensive meta-information on the included datasets; and (c) allow benchmarks to be shared and reused in future studies. We then present a first, carefully curated and practical benchmarking suite for classification: the OpenML Curated Classification benchmarking suite 2018 (OpenML-CC18). Finally, we discuss use cases and applications which demonstrate the usefulness of OpenML benchmarking suites and the OpenML-CC18 in particular.