LGAR: Zero-Shot LLM-Guided Neural Ranking for Abstract Screening in Systematic Literature Reviews

TL;DR

This paper introduces LGAR, a zero-shot LLM-guided neural ranking method for abstract screening in systematic literature reviews, outperforming existing QA-based approaches in precision and relevance evaluation.

Contribution

The paper presents LGAR, a novel zero-shot neural ranking framework guided by large language models, with a comprehensive benchmark and extraction of criteria for fair comparison.

Findings

LGAR outperforms existing QA-based methods by 5-10 percentage points in mean average precision.

Manual extraction of criteria enables fair and detailed comparison of approaches.

Extensive experiments validate the effectiveness of LGAR in systematic literature review tasks.

Abstract

The scientific literature is growing rapidly, making it hard to keep track of the state-of-the-art. Systematic literature reviews (SLRs) aim to identify and evaluate all relevant papers on a topic. After retrieving a set of candidate papers, the abstract screening phase determines initial relevance. To date, abstract screening methods using large language models (LLMs) focus on binary classification settings; existing question answering (QA) based ranking approaches suffer from error propagation. LLMs offer a unique opportunity to evaluate the SLR's inclusion and exclusion criteria, yet, existing benchmarks do not provide them exhaustively. We manually extract these criteria as well as research questions for 57 SLRs, mostly in the medical domain, enabling principled comparisons between approaches. Moreover, we propose LGAR, a zero-shot LLM Guided Abstract Ranker composed of an LLM based graded relevance scorer and a dense re-ranker. Our extensive experiments show that LGAR outperforms existing QA-based methods by 5-10 pp. in mean average precision. Our code and data is publicly available.