LLMClean: Context-Aware Tabular Data Cleaning via LLM-Generated OFDs

TL;DR

LLMClean leverages large language models to automatically generate context models for tabular data cleaning, achieving results comparable to expert-crafted models across diverse datasets.

Contribution

This paper presents a novel automated method using LLMs to generate context models for data cleaning, reducing reliance on domain experts.

Findings

Automated context models perform comparably to human-crafted models.

Effective across datasets from IoT, healthcare, and Industry 4.0.

Demonstrates potential for scalable, resource-efficient data cleaning.

Abstract

Machine learning's influence is expanding rapidly, now integral to decision-making processes from corporate strategy to the advancements in Industry 4.0. The efficacy of Artificial Intelligence broadly hinges on the caliber of data used during its training phase; optimal performance is tied to exceptional data quality. Data cleaning tools, particularly those that exploit functional dependencies within ontological frameworks or context models, are instrumental in augmenting data quality. Nevertheless, crafting these context models is a demanding task, both in terms of resources and expertise, often necessitating specialized knowledge from domain experts. In light of these challenges, this paper introduces an innovative approach, called LLMClean, for the automated generation of context models, utilizing Large Language Models to analyze and understand various datasets. LLMClean encompasses a sequence of actions, starting with categorizing the dataset, extracting or mapping relevant models, and ultimately synthesizing the context model. To demonstrate its potential, we have developed and tested a prototype that applies our approach to three distinct datasets from the Internet of Things, healthcare, and Industry 4.0 sectors. The results of our evaluation indicate that our automated approach can achieve data cleaning efficacy comparable with that of context models crafted by human experts.