Expediting data extraction using a large language model (LLM) and scoping review protocol: a methodological study within a complex scoping review

TL;DR

This study explores using large language models with review protocols to expedite data extraction in complex scoping reviews, highlighting high accuracy for simple data but challenges with complex, subjective data items.

Contribution

It demonstrates the potential and limitations of LLMs in data extraction within scoping reviews and emphasizes the need for robust evaluation and reporting of LLM performance.

Findings

High accuracy for simple citation data extraction (83.3%-100%)

Lower accuracy for complex, subjective data (9.6%-15.8%)

LLM feedback can aid protocol refinement

Abstract

The data extraction stages of reviews are resource-intensive, and researchers may seek to expediate data extraction using online (large language models) LLMs and review protocols. Claude 3.5 Sonnet was used to trial two approaches that used a review protocol to prompt data extraction from 10 evidence sources included in a case study scoping review. A protocol-based approach was also used to review extracted data. Limited performance evaluation was undertaken which found high accuracy for the two extraction approaches (83.3% and 100%) when extracting simple, well-defined citation details; accuracy was lower (9.6% and 15.8%) when extracting more complex, subjective data items. Considering all data items, both approaches had precision >90% but low recall (<25%) and F1 scores (<40%). The context of a complex scoping review, open response types and methodological approach likely impacted performance due to missed and misattributed data. LLM feedback considered the baseline extraction accurate and suggested minor amendments: four of 15 (26.7%) to citation details and 8 of 38 (21.1%) to key findings data items were considered to potentially add value. However, when repeating the process with a dataset featuring deliberate errors, only 2 of 39 (5%) errors were detected. Review-protocol-based methods used for expediency require more robust performance evaluation across a range of LLMs and review contexts with comparison to conventional prompt engineering approaches. We recommend researchers evaluate and report LLM performance if using them similarly to conduct data extraction or review extracted data. LLM feedback contributed to protocol adaptation and may assist future review protocol drafting.