Nonstandard Errors in AI Agents

TL;DR

This paper investigates the variability in results produced by AI coding agents when analyzing the same data, revealing significant nonstandard errors and systematic differences in methodological choices, with implications for AI-driven research reliability.

Contribution

It introduces the concept of nonstandard errors in AI agents, demonstrating how methodological divergence affects empirical results and exploring how peer review influences convergence.

Findings

AI agents show substantial variation in analysis choices.

Different model families have stable, systematic differences.

Peer review has limited impact on reducing variability.

Abstract

We study whether state-of-the-art AI coding agents, given the same data and research question, produce the same empirical results. Deploying 150 autonomous Claude Code agents to independently test six hypotheses about market quality trends in NYSE TAQ data for SPY (2015--2024), we find that AI agents exhibit sizable \textit{nonstandard errors} (NSEs), that is, uncertainty from agent-to-agent variation in analytical choices, analogous to those documented among human researchers. AI agents diverge substantially on measure choice (e.g., autocorrelation vs.\ variance ratio, dollar vs.\ share volume). Different model families (Sonnet 4.6 vs.\ Opus 4.6) exhibit stable ``empirical styles,'' reflecting systematic differences in methodological preferences. In a three-stage feedback protocol, AI peer review (written critiques) has minimal effect on dispersion, whereas exposure to top-rated exemplar papers reduces the interquartile range of estimates by 80--99\% within \textit{converging} measure families. Convergence occurs both through within-family estimation tightening and through agents switching measure families entirely, but convergence reflects imitation rather than understanding. These findings have implications for the growing use of AI in automated policy evaluation and empirical research.