MetaLead: A Comprehensive Human-Curated Leaderboard Dataset for Transparent Reporting of Machine Learning Experiments

TL;DR

MetaLead is a detailed, human-curated ML leaderboard dataset that captures all experimental results and metadata, enabling more transparent and nuanced evaluation of machine learning research progress.

Contribution

It introduces MetaLead, a comprehensive dataset that includes all experimental results and metadata, improving transparency and evaluation in ML benchmarking.

Findings

Includes all experimental results for transparency

Contains metadata like experiment type and dataset splits

Enables cross-domain assessment

Abstract

Leaderboards are crucial in the machine learning (ML) domain for benchmarking and tracking progress. However, creating leaderboards traditionally demands significant manual effort. In recent years, efforts have been made to automate leaderboard generation, but existing datasets for this purpose are limited by capturing only the best results from each paper and limited metadata. We present MetaLead, a fully human-annotated ML Leaderboard dataset that captures all experimental results for result transparency and contains extra metadata, such as the result experimental type: baseline, proposed method, or variation of proposed method for experiment-type guided comparisons, and explicitly separates train and test dataset for cross-domain assessment. This enriched structure makes MetaLead a powerful resource for more transparent and nuanced evaluations across ML research.