Certifying One-Phase Technology-Assisted Reviews

TL;DR

This paper introduces statistically valid stopping rules for one-phase technology-assisted review workflows, addressing a key gap and demonstrating that controlling recall overshoot reduces overall review costs.

Contribution

It provides the first broadly applicable, statistically valid sample-based stopping rules for one-phase TAR workflows based on quantile estimation.

Findings

New stopping rules with statistical guarantees

Overshooting recall increases total review costs

Reducing recall overshoot lowers overall costs

Abstract

Technology-assisted review (TAR) workflows based on iterative active learning are widely used in document review applications. Most stopping rules for one-phase TAR workflows lack valid statistical guarantees, which has discouraged their use in some legal contexts. Drawing on the theory of quantile estimation, we provide the first broadly applicable and statistically valid sample-based stopping rules for one-phase TAR. We further show theoretically and empirically that overshooting a recall target, which has been treated as innocuous or desirable in past evaluations of stopping rules, is a major source of excess cost in one-phase TAR workflows. Counterintuitively, incurring a larger sampling cost to reduce excess recall leads to lower total cost in almost all scenarios.