Evaluating Sentence-Level Relevance Feedback for High-Recall Information Retrieval

TL;DR

This paper introduces a simulation framework to assess if providing isolated sentences for relevance feedback improves high-recall information retrieval efficiency, showing comparable accuracy with less effort compared to traditional document review.

Contribution

It demonstrates that sentence-level relevance feedback can achieve similar recall with higher efficiency, offering a novel approach to active learning in information retrieval.

Findings

Sentence-level feedback yields comparable recall to document-level.

Using isolated sentences reduces review effort.

Method outperforms the baseline CAL approach in simulations.

Abstract

This study uses a novel simulation framework to evaluate whether the time and effort necessary to achieve high recall using active learning is reduced by presenting the reviewer with isolated sentences, as opposed to full documents, for relevance feedback. Under the weak assumption that more time and effort is required to review an entire document than a single sentence, simulation results indicate that the use of isolated sentences for relevance feedback can yield comparable accuracy and higher efficiency, relative to the state-of-the-art Baseline Model Implementation (BMI) of the AutoTAR Continuous Active Learning ("CAL") method employed in the TREC 2015 and 2016 Total Recall Track.