Principles and Guidelines for Randomized Controlled Trials in AI Evaluation

TL;DR

This paper develops a comprehensive framework for standardizing AI evaluation through randomized controlled trials, emphasizing transparency, human-centered assessment, and addressing AI-specific challenges.

Contribution

It introduces a set of 33 guidelines based on established RCT principles, adapted for AI evaluation, serving as a design tool, assessment rubric, and standardization blueprint.

Findings

Operationalized five core principles into practical guidelines.

Centered evaluation on human performance rather than model output.

Addressed AI-specific challenges like model versioning and spillover effects.

Abstract

This work establishes a foundational framework for standardizing AI evaluation RCTs (sometimes called human uplift studies). Drawing on established experimental practices from disciplines with established RCT traditions, including software engineering, economics, clinical and health sciences, and psychology, we adopt the (Shadish et al., 2002) four-validity framework and extend it with a fifth principle on transparency, repeatability, and verification adapted from the Transparency and Openness Promotion (TOP) Guidelines (Center for Open Science, 2025). We operationalize all five principles into 33 guidelines adapted for AI evaluation RCT contexts, expressed as requirements with rationales, implementation instructions, and evidence bases. We position the principles and guidelines as serving three key roles for AI evaluation RCTs: a design tool for planning studies, an evaluation rubric for assessing existing work, and a blueprint for standard setting as the field converges on norms. Our framework extends prior work by centering evaluation on human performance rather than model output alone, formalizing causal inference through RCT methodology for AI contexts, integrating heterogeneity analysis and practical significance assessment, implementing a graded transparency and repeatability framework, and addressing AI-specific challenges including model versioning, human-AI interaction dynamics, contamination and spillover effects, and equitable impact assessment.