Neural Rankers for Effective Screening Prioritisation in Medical Systematic Review Literature Search

TL;DR

This paper explores the application of pre-trained language models, especially BERT, to improve document ranking in medical systematic review screening, demonstrating superior performance over traditional methods and potential for combined approaches.

Contribution

It introduces the use of pre-trained language models for systematic review screening prioritisation and compares their effectiveness to traditional ranking methods.

Findings

BERT-based rankers outperform existing screening prioritisation methods.

Neural models and traditional methods can be complementary.

Different document representations impact neural ranking performance.

Abstract

Medical systematic reviews typically require assessing all the documents retrieved by a search. The reason is two-fold: the task aims for ``total recall''; and documents retrieved using Boolean search are an unordered set, and thus it is unclear how an assessor could examine only a subset. Screening prioritisation is the process of ranking the (unordered) set of retrieved documents, allowing assessors to begin the downstream processes of the systematic review creation earlier, leading to earlier completion of the review, or even avoiding screening documents ranked least relevant. Screening prioritisation requires highly effective ranking methods. Pre-trained language models are state-of-the-art on many IR tasks but have yet to be applied to systematic review screening prioritisation. In this paper, we apply several pre-trained language models to the systematic review document ranking task, both directly and fine-tuned. An empirical analysis compares how effective neural methods compare to traditional methods for this task. We also investigate different types of document representations for neural methods and their impact on ranking performance. Our results show that BERT-based rankers outperform the current state-of-the-art screening prioritisation methods. However, BERT rankers and existing methods can actually be complementary, and thus, further improvements may be achieved if used in conjunction.