On Minimizing Cost in Legal Document Review Workflows

TL;DR

This paper analyzes how the relative costs of manual labeling influence the choice between one-phase and two-phase technology-assisted review workflows in legal document review, providing analytical and empirical insights.

Contribution

It introduces a novel cost dynamics analysis to determine optimal review workflows based on cost factors, classification difficulty, and collection size.

Findings

Cost dynamics significantly influence workflow choice.

Two-phase workflows can be more cost-effective under certain cost conditions.

Optimal active learning methods depend on document prevalence and task difficulty.

Abstract

Technology-assisted review (TAR) refers to human-in-the-loop machine learning workflows for document review in legal discovery and other high recall review tasks. Attorneys and legal technologists have debated whether review should be a single iterative process (one-phase TAR workflows) or whether model training and review should be separate (two-phase TAR workflows), with implications for the choice of active learning algorithm. The relative cost of manual labeling for different purposes (training vs. review) and of different documents (positive vs. negative examples) is a key and neglected factor in this debate. Using a novel cost dynamics analysis, we show analytically and empirically that these relative costs strongly impact whether a one-phase or two-phase workflow minimizes cost. We also show how category prevalence, classification task difficulty, and collection size impact the optimal choice not only of workflow type, but of active learning method and stopping point.