Generating Natural Language Queries for More Effective Systematic Review Screening Prioritisation

TL;DR

This paper investigates alternative query sources for prioritising document screening in systematic reviews, demonstrating that generative language models can produce effective queries comparable to final review titles, improving early-stage screening efficiency.

Contribution

It introduces the use of instruction-based generative language models to create queries for screening prioritisation, addressing the impracticality of relying on final review titles.

Findings

Generated queries from language models perform similarly to final titles.

Using Boolean retrieval queries improves early screening prioritisation.

Language models provide viable query sources at the initial screening stage.

Abstract

Screening prioritisation in medical systematic reviews aims to rank the set of documents retrieved by complex Boolean queries. Prioritising the most important documents ensures that subsequent review steps can be carried out more efficiently and effectively. The current state of the art uses the final title of the review as a query to rank the documents using BERT-based neural rankers. However, the final title is only formulated at the end of the review process, which makes this approach impractical as it relies on ex post facto information. At the time of screening, only a rough working title is available, with which the BERT-based ranker performs significantly worse than with the final title. In this paper, we explore alternative sources of queries for prioritising screening, such as the Boolean query used to retrieve the documents to be screened and queries generated by instruction-based generative large-scale language models such as ChatGPT and Alpaca. Our best approach is not only viable based on the information available at the time of screening, but also has similar effectiveness to the final title.