CleanAgent: Automating Data Standardization with LLM-based Agents

TL;DR

CleanAgent is a framework that automates data standardization using LLM-based agents and a Python library, simplifying the process for data scientists and reducing manual coding effort.

Contribution

It introduces a unified API library and an LLM-integrated framework that automates data standardization, reducing complexity and manual intervention.

Findings

Significantly reduces coding complexity for data standardization.

Enables hands-free, requirement-based automation with LLM agents.

Provides a user-friendly web application for practical use.

Abstract

Data standardization is a crucial part of the data science life cycle. While tools like Pandas offer robust functionalities, their complexity and the manual effort required for customizing code to diverse column types pose significant challenges. Although large language models (LLMs) like ChatGPT have shown promise in automating this process through natural language understanding and code generation, it still demands expert-level programming knowledge and continuous interaction for prompt refinement. To solve these challenges, our key idea is to propose a Python library with declarative, unified APIs for standardizing different column types, simplifying the LLM's code generation with concise API calls. We first propose Dataprep.Clean, a component of the Dataprep Python Library, significantly reduces the coding complexity by enabling the standardization of specific column types with a single line of code. Then, we introduce the CleanAgent framework integrating Dataprep.Clean and LLM-based agents to automate the data standardization process. With CleanAgent, data scientists only need to provide their requirements once, allowing for a hands-free process. To demonstrate the practical utility of CleanAgent, we developed a user-friendly web application, allowing users to interact with it using real-world datasets.