RLStop: A Reinforcement Learning Stopping Method for TAR

TL;DR

RLStop is a reinforcement learning-based stopping rule for TAR that minimizes manual review workload while maintaining high recall, outperforming existing methods across multiple benchmark datasets.

Contribution

Introduces RLStop, a novel reinforcement learning approach for TAR stopping that adapts to different datasets and target recall levels, improving efficiency.

Findings

Reduces manual review workload significantly.

Achieves near-optimal performance compared to other methods.

Effective across diverse benchmark datasets.

Abstract

We present RLStop, a novel Technology Assisted Review (TAR) stopping rule based on reinforcement learning that helps minimise the number of documents that need to be manually reviewed within TAR applications. RLStop is trained on example rankings using a reward function to identify the optimal point to stop examining documents. Experiments at a range of target recall levels on multiple benchmark datasets (CLEF e-Health, TREC Total Recall, and Reuters RCV1) demonstrated that RLStop substantially reduces the workload required to screen a document collection for relevance. RLStop outperforms a wide range of alternative approaches, achieving performance close to the maximum possible for the task under some circumstances.