Text-to-Pipeline: Bridging Natural Language and Data Preparation Pipelines

TL;DR

Text-to-Pipeline introduces a task translating natural language data prep instructions into executable pipelines, supported by a large benchmark, revealing current LLM limitations and proposing an iterative agent baseline.

Contribution

The paper presents the novel Text-to-Pipeline task, a large-scale benchmark PARROT, and an execution-aware agent baseline to advance automated data preparation from natural language instructions.

Findings

LLMs struggle with multi-step compositional logic.

Semantic parameter grounding remains a challenge for LLMs.

Pipeline-Agent achieves state-of-the-art performance but still has significant gaps.

Abstract

Data preparation (DP) transforms raw data into a form suitable for downstream applications, typically by composing operations into executable pipelines. Building such pipelines is time-consuming and requires sophisticated programming skills, posing a significant barrier for non-experts. To lower this barrier, we introduce Text-to-Pipeline, a new task that translates NL data preparation instructions into DP pipelines, and PARROT, a large-scale benchmark to support systematic evaluation. To ensure realistic DP scenarios, PARROT is built by mining transformation patterns from production pipelines and instantiating them on 23,009 real-world tables, resulting in ~18,000 tasks spanning 16 core operators. Our empirical evaluation on PARROT reveals a critical failure mode in cutting-edge LLMs: they struggle not only with multi-step compositional logic but also with semantic parameter grounding. We thus establish a strong baseline with Pipeline-Agent, an execution-aware agent that iteratively reflects on intermediate states. While it achieves state-of-the-art performance, a significant gap remains, underscoring the deep, unsolved challenges for PARROT. It provides the essential, large-scale testbed for developing and evaluating the next generation of autonomous data preparation agentic systems.