PrepBench: How Far Are We from Natural-Language-Driven Data Preparation?

TL;DR

PrepBench is a new benchmark designed to evaluate how well current large language models can support natural language-driven data preparation tasks, highlighting existing challenges and gaps.

Contribution

The paper introduces PrepBench, a comprehensive benchmark for assessing LLM capabilities in NL-driven data prep, addressing key real-world complexities.

Findings

Current LLMs struggle with complex, multi-step data preparation tasks.

Nearly half of the tasks require over 100 lines of Python code.

Realizing NL-driven data preparation remains a significant challenge for state-of-the-art models.

Abstract

Data preparation is a central and time-consuming stage in data analysis workflows. Traditionally, commercial tools have relied on graphical user interfaces (GUIs) to simplify data preparation, allowing users to define transformations through visual operators and workflows. Recent advances in large language models (LLMs) raise the possibility of a paradigm shift toward natural language (NL)-driven data preparation, in which users can specify preparation intents in NL directly. However, it remains unclear how far current LLM-based agents are from this paradigm shift in practice. Existing code generation benchmarks do not capture key characteristics of data preparation, including ambiguous user intents, imperfect real-world data, and the need to translate code into interpretable workflows for validation. To bridge this gap, we present PrepBench, a benchmark designed to evaluate NL-driven data preparation along three core capabilities: interactive disambiguation, prep-code generation, and code-to-workflow translation. We crawl data from the Preppin' Data Challenges, and then extend it into a systematically designed benchmark. The benchmark covers diverse domains, and each task involves 3 to 18 data preparation steps. Nearly half of the tasks require over 100 lines of Python code, and the longest solutions approach 300 lines. Our evaluation shows that, despite recent progress, realizing this paradigm shift remains challenging for state-of-the-art LLMs. PrepBench provides a principled benchmark for measuring this gap and helps identify key challenges toward realizing NL-driven data preparation.