Goldilocks: Just-Right Tuning of BERT for Technology-Assisted Review

TL;DR

This paper evaluates the effectiveness of BERT, a transformer-based model, in technology-assisted review workflows, finding that domain-specific fine-tuning is crucial for optimal performance, with mixed results across different datasets.

Contribution

It demonstrates the importance of task-specific fine-tuning of BERT in TAR workflows and highlights the impact of domain match on model effectiveness.

Findings

BERT reduces review costs by 10-15% on RCV1-v2.

Linear models outperform BERT on legal discovery datasets.

Proper fine-tuning is critical for BERT's success in TAR.

Abstract

Technology-assisted review (TAR) refers to iterative active learning workflows for document review in high recall retrieval (HRR) tasks. TAR research and most commercial TAR software have applied linear models such as logistic regression to lexical features. Transformer-based models with supervised tuning are known to improve effectiveness on many text classification tasks, suggesting their use in TAR. We indeed find that the pre-trained BERT model reduces review cost by 10% to 15% in TAR workflows simulated on the RCV1-v2 newswire collection. In contrast, we likewise determined that linear models outperform BERT for simulated legal discovery topics on the Jeb Bush e-mail collection. This suggests the match between transformer pre-training corpora and the task domain is of greater significance than generally appreciated. Additionally, we show that just-right language model fine-tuning on the task collection before starting active learning is critical. Too little or too much fine-tuning hinders performance, worse than that of linear models, even for a favorable corpus such as RCV1-v2.