AIRepr: An Analyst-Inspector Framework for Evaluating Reproducibility of LLMs in Data Science

TL;DR

AIRepr is a framework that automatically evaluates and enhances the reproducibility of LLM-generated data analysis workflows, improving reliability and accuracy in automated data science tasks.

Contribution

The paper introduces AIRepr, a novel framework with reproducibility-enhancing prompts for assessing and improving LLM-generated workflows in data analysis.

Findings

Reproducible workflows lead to more accurate analyses.

Reproducibility-enhancing prompts outperform standard prompts.

Framework validated across multiple benchmarks and LLM pairs.

Abstract

Large language models (LLMs) are increasingly used to automate data analysis through executable code generation. Yet, data science tasks often admit multiple statistically valid solutions, e.g. different modeling strategies, making it critical to understand the reasoning behind analyses, not just their outcomes. While manual review of LLM-generated code can help ensure statistical soundness, it is labor-intensive and requires expertise. A more scalable approach is to evaluate the underlying workflows-the logical plans guiding code generation. However, it remains unclear how to assess whether an LLM-generated workflow supports reproducible implementations. To address this, we present AIRepr, an Analyst-Inspector framework for automatically evaluating and improving the reproducibility of LLM-generated data analysis workflows. Our framework is grounded in statistical principles and supports scalable, automated assessment. We introduce two novel reproducibility-enhancing prompting strategies and benchmark them against standard prompting across 15 analyst-inspector LLM pairs and 1,032 tasks from three public benchmarks. Our findings show that workflows with higher reproducibility also yield more accurate analyses, and that reproducibility-enhancing prompts substantially improve both metrics. This work provides a foundation for transparent, reliable, and efficient human-AI collaboration in data science. Our code is publicly available.