DataPerf: Benchmarks for Data-Centric AI Development

TL;DR

DataPerf introduces a comprehensive benchmark suite for evaluating and advancing data-centric AI, promoting innovation, reproducibility, and community collaboration across diverse data tasks and modalities.

Contribution

It presents the first community-led, open-source benchmark platform specifically designed for data-centric AI, enabling iterative dataset development and comparison.

Findings

Contains five diverse benchmarks covering vision, speech, and more.

Supports multiple rounds of community challenges and contributions.

Open-source platform with baseline implementations for reproducibility.

Abstract

Machine learning research has long focused on models rather than datasets, and prominent datasets are used for common ML tasks without regard to the breadth, difficulty, and faithfulness of the underlying problems. Neglecting the fundamental importance of data has given rise to inaccuracy, bias, and fragility in real-world applications, and research is hindered by saturation across existing dataset benchmarks. In response, we present DataPerf, a community-led benchmark suite for evaluating ML datasets and data-centric algorithms. We aim to foster innovation in data-centric AI through competition, comparability, and reproducibility. We enable the ML community to iterate on datasets, instead of just architectures, and we provide an open, online platform with multiple rounds of challenges to support this iterative development. The first iteration of DataPerf contains five benchmarks covering a wide spectrum of data-centric techniques, tasks, and modalities in vision, speech, acquisition, debugging, and diffusion prompting, and we support hosting new contributed benchmarks from the community. The benchmarks, online evaluation platform, and baseline implementations are open source, and the MLCommons Association will maintain DataPerf to ensure long-term benefits to academia and industry.