DCA-Bench: A Benchmark for Dataset Curation Agents

TL;DR

This paper introduces DCA-Bench, a benchmark for evaluating dataset curation agents using real-world test cases and an automatic GPT-4o-based evaluation, highlighting the challenges faced by LLMs in detecting data quality issues.

Contribution

The work establishes a benchmark with curated test cases and an evaluation framework to assess LLM agents' ability to identify dataset issues in real-world scenarios.

Findings

Most LLM agents detect only about 30% of data issues without hints.

The benchmark reveals the complexity of dataset curation for LLMs.

GPT-4o evaluation aligns well with expert assessments.

Abstract

The quality of datasets plays an increasingly crucial role in the research and development of modern artificial intelligence (AI). Despite the proliferation of open dataset platforms nowadays, data quality issues, such as incomplete documentation, inaccurate labels, ethical concerns, and outdated information, remain common in widely used datasets. Furthermore, these issues are often subtle and difficult to be detected by rule-based scripts, therefore requiring identification and verification by dataset users or maintainers--a process that is both time-consuming and prone to human mistakes. With the surging ability of large language models (LLM), it's promising to streamline the discovery of hidden dataset issues with LLM agents. To achieve this, one significant challenge is enabling LLM agents to detect issues in the wild rather than simply fixing known ones. In this work, we establish a benchmark to measure LLM agent's ability to tackle this challenge. We carefully curate 221 real-world test cases from eight popular dataset platforms and propose an automatic evaluation framework using GPT-4o. Our proposed framework shows strong empirical alignment with expert evaluations, validated through extensive comparisons with human annotations. Without any hints, most competitive Curator agent can only reveal $\sim$30\% of the data quality issues in the proposed dataset, highlighting the complexity of this task and indicating that applying LLM agents to real-world dataset curation still requires further in-depth exploration and innovation. The data and code are available at \href{https://github.com/TRAIS-Lab/dca-bench}{https://github.com/TRAIS-Lab/dca-bench}.