Cutting Through the Clutter: The Potential of LLMs for Efficient Filtration in Systematic Literature Reviews

TL;DR

This paper demonstrates that Large Language Models can significantly reduce the time and improve the accuracy of literature filtering in systematic reviews through an interactive, open-source tool, enhancing research efficiency.

Contribution

Introduces LLMSurver, an open-source tool leveraging LLMs for efficient literature filtration with interactive query refinement and high recall, advancing systematic review methodologies.

Findings

LLMs reduce filtering time from weeks to minutes.

Recall rates exceed 98.8%, surpassing human error thresholds.

The approach improves accuracy and efficiency in literature reviews.

Abstract

Systematic literature reviews (SLRs) are essential but labor-intensive due to high publication volumes and inefficient keyword-based filtering. To streamline this process, we evaluate Large Language Models (LLMs) for enhancing efficiency and accuracy in corpus filtration while minimizing manual effort. Our open-source tool LLMSurver presents a visual interface to utilize LLMs for literature filtration, evaluate the results, and refine queries in an interactive way. We assess the real-world performance of our approach in filtering over 8.3k articles during a recent survey construction, comparing results with human efforts. The findings show that recent LLM models can reduce filtering time from weeks to minutes. A consensus scheme ensures recall rates >98.8%, surpassing typical human error thresholds and improving selection accuracy. This work advances literature review methodologies and highlights the potential of responsible human-AI collaboration in academic research.