Sanity Checks for Agentic Data Science

TL;DR

This paper introduces lightweight sanity checks based on the PCS framework to evaluate the trustworthiness of agentic data science systems like OpenAI Codex, ensuring their conclusions are reliable and not noise-driven.

Contribution

It proposes a novel set of sanity checks grounded in PCS to assess the stability and reliability of ADS outputs, validated on synthetic and real datasets.

Findings

Sanity checks effectively track ground-truth signal strength in synthetic data.

In 6 out of 11 real datasets, affirmative conclusions were unsupported by the sanity checks.

ADS confidence is poorly calibrated to the stability of its conclusions.

Abstract

Agentic data science (ADS) pipelines have grown rapidly in both capability and adoption, with systems such as OpenAI Codex now able to directly analyze datasets and produce answers to statistical questions. However, these systems can reach falsely optimistic conclusions that are difficult for users to detect. To address this, we propose a pair of lightweight sanity checks grounded in the Predictability-Computability-Stability (PCS) framework for veridical data science. These checks use reasonable perturbations to screen whether an agent can reliably distinguish signal from noise, acting as a falsifiability constraint that can expose affirmative conclusions as unsupported. Together, the two checks characterize the trustworthiness of an ADS output, e.g. whether it has found stable signal, is responding to noise, or is sensitive to incidental aspects of the input. We validate the approach on synthetic data with controlled signal-to-noise ratios, confirming that the sanity checks track ground-truth signal strength. We then demonstrate the checks on 11 real-world datasets using OpenAI Codex, characterizing the trustworthiness of each conclusion and finding that in 6 of the datasets an affirmative conclusion is not well-supported, even though a single ADS run may support one. We further analyze failure modes of ADS systems and find that ADS self-reported confidence is poorly calibrated to the empirical stability of its conclusions.