ViseGPT: Towards Better Alignment of LLM-generated Data Wrangling Scripts and User Prompts

TL;DR

ViseGPT is a tool that automatically generates test cases from user prompts to verify and improve the correctness of LLM-generated data wrangling scripts, streamlining debugging and user workflow.

Contribution

We introduce ViseGPT, a novel system that extracts constraints from prompts to create test cases, improving validation and debugging of LLM-generated scripts.

Findings

ViseGPT significantly improves debugging efficiency.

Users better detect and correct script issues.

Streamlines the data wrangling workflow.

Abstract

Large language models (LLMs) enable the rapid generation of data wrangling scripts based on natural language instructions, but these scripts may not fully adhere to user-specified requirements, necessitating careful inspection and iterative refinement. Existing approaches primarily assist users in understanding script logic and spotting potential issues themselves, rather than providing direct validation of correctness. To enhance debugging efficiency and optimize the user experience, we develop ViseGPT, a tool that automatically extracts constraints from user prompts to generate comprehensive test cases for verifying script reliability. The test results are then transformed into a tailored Gantt chart, allowing users to intuitively assess alignment with semantic requirements and iteratively refine their scripts. Our design decisions are informed by a formative study (N=8) that explores user practices and challenges. We further evaluate the effectiveness and usability of ViseGPT through a user study (N=18). Results indicate that ViseGPT significantly improves debugging efficiency for LLM-generated data-wrangling scripts, enhances users' ability to detect and correct issues, and streamlines the workflow experience.