Technology Assisted Reviews: Finding the Last Few Relevant Documents by Asking Yes/No Questions to Reviewers

TL;DR

This paper introduces a Bayesian questioning method to efficiently find the last relevant documents in a collection, reducing human effort in technology-assisted reviews.

Contribution

It proposes a novel sequential Bayesian search approach that uses yes/no questions about entities to identify remaining relevant documents more efficiently.

Findings

Significantly reduces review effort for last relevant documents

Outperforms existing active learning methods in experiments

Demonstrates effectiveness in real-world datasets

Abstract

The goal of a technology-assisted review is to achieve high recall with low human effort. Continuous active learning algorithms have demonstrated good performance in locating the majority of relevant documents in a collection, however their performance is reaching a plateau when 80\%-90\% of them has been found. Finding the last few relevant documents typically requires exhaustively reviewing the collection. In this paper, we propose a novel method to identify these last few, but significant, documents efficiently. Our method makes the hypothesis that entities carry vital information in documents, and that reviewers can answer questions about the presence or absence of an entity in the missing relevance documents. Based on this we devise a sequential Bayesian search method that selects the optimal sequence of questions to ask. The experimental results show that our proposed method can greatly improve performance requiring less reviewing effort.