Generative and reproducible benchmarks for comprehensive evaluation of machine learning classifiers

TL;DR

This paper introduces DIGEN, a comprehensive collection of synthetic datasets generated from diverse mathematical functions, enabling reproducible and interpretable benchmarking of machine learning classifiers for binary outcomes.

Contribution

The paper presents DIGEN, a novel benchmark suite of 40 synthetic datasets created using heuristic-discovered functions to evaluate ML classifiers comprehensively and reproducibly.

Findings

Provides a diverse set of synthetic datasets for benchmarking.

Facilitates understanding of algorithm performance differences.

Open-source resource with extensive documentation.

Abstract

Understanding the strengths and weaknesses of machine learning (ML) algorithms is crucial for determine their scope of application. Here, we introduce the DIverse and GENerative ML Benchmark (DIGEN) - a collection of synthetic datasets for comprehensive, reproducible, and interpretable benchmarking of machine learning algorithms for classification of binary outcomes. The DIGEN resource consists of 40 mathematical functions which map continuous features to discrete endpoints for creating synthetic datasets. These 40 functions were discovered using a heuristic algorithm designed to maximize the diversity of performance among multiple popular machine learning algorithms thus providing a useful test suite for evaluating and comparing new methods. Access to the generative functions facilitates understanding of why a method performs poorly compared to other algorithms thus providing ideas for improvement. The resource with extensive documentation and analyses is open-source and available on GitHub.