Chained Prompting for Better Systematic Review Search Strategies

TL;DR

This paper presents an LLM-based chained prompt framework that automates the development of systematic review search strategies, achieving high recall and outperforming existing methods in retrieval effectiveness.

Contribution

It introduces a novel LLM-driven approach that automates search strategy design, including PICO extraction and Boolean query synthesis, improving efficiency and recall in systematic reviews.

Findings

Achieves an average recall of 0.9 on LEADSInstruct dataset

Outperforms existing search strategy methods in recall

Highlights importance of precise objective specification

Abstract

Systematic reviews require the use of rigorously designed search strategies to ensure both comprehensive retrieval and minimization of bias. Conventional manual approaches, although methodologically systematic, are resource-intensive and susceptible to subjectivity, whereas heuristic and automated techniques frequently under-perform in recall unless supplemented by extensive expert input. We introduce a Large Language Model (LLM)-based chained prompt engineering framework for the automated development of search strategies in systematic reviews. The framework replicates the procedural structure of manual search design while leveraging LLMs to decompose review objectives, extract and formalize PICO elements, generate conceptual representations, expand terminologies, and synthesize Boolean queries. In addition to query construction, the framework exhibits superior performance in generating well-structured PICO elements relative to existing methods, thereby strengthening the foundation for high-recall search strategies. Evaluation on a subset of the LEADSInstruct dataset demonstrates that the framework attains a 0.9 average recall. These results significantly exceed the performance of existing approaches. Error analysis further highlights the critical role of precise objective specification and terminological alignment in optimizing retrieval effectiveness. These findings confirm the capacity of LLM-based pipelines to yield transparent, reproducible, and high-performing search strategies, and highlight their potential as scalable instruments for supporting evidence synthesis and evidence-based practice.