Towards Practical Benchmarking of Data Cleaning Techniques: On Generating Authentic Errors via Large Language Models

TL;DR

This paper presents TableEG, a framework using large language models to generate authentic, diverse errors in tabular data, enabling more realistic benchmarking of data cleaning techniques.

Contribution

The paper introduces TableEG, a novel LLM-based approach for generating realistic errors in tabular data, improving evaluation of error detection and correction methods.

Findings

Errors generated by TableEG closely mimic real-world error distributions.

Performance on TableEG errors aligns with real-world error detection results.

TableEG outperforms rule-based and non-fine-tuned LLM methods in error realism.

Abstract

Data quality remains an important challenge in data-driven systems, as errors in tabular data can severely compromise downstream analytics and machine learning performance. Although numerous error detection algorithms have been proposed, the lack of diverse, real-world error datasets limits comprehensive evaluation. Manual error annotation is both time-consuming and inconsistent, motivating the exploration of synthetic error generation as an alternative. In this work, we introduce TableEG, a framework that leverages large language models (LLMs) to generate authentic errors. By employing a table fine-tuning strategy and a triplet representation $(I, T, O)$ to model error generation, detection, and correction tasks, TableEG captures the complex dependencies inherent in two-dimensional tables. Trained on 12 real-world datasets spanning 10 diverse domains, TableEG ensures that the synthesized errors faithfully reflect authentic error distributions. Experimental results indicate that errors generated by TableEG exhibit superior pattern and distribution similarity compared to both rule-based methods and LLM-generated errors without fine-tuning. Furthermore, performance metrics on TableEG-generated errors closely align with those on real-world errors across nearly all datasets and detection algorithms, particularly for machine learning based detection techniques. Overall, TableEG not only bridges the gap between synthetic and real-world errors but also establishes a robust benchmark for subsequent error detection and correction tasks.