Human researchers are superior to large language models in writing a medical systematic review in a comparative multitask assessment

TL;DR

This study compares human researchers and large language models in performing a medical systematic review and finds that humans outperform LLMs in most tasks.

Contribution

The study provides a comparative multitask assessment of LLMs versus humans in medical systematic reviews.

Findings

The best LLM identified 13 out of 18 relevant articles in the literature search task.

LLMs struggled with accurate data extraction and analysis.

LLM-generated papers were short and did not fully follow the PRISMA 2020 template.

Abstract

The capability of Large Language Models (LLMs) to support and facilitate research activities has sparked growing interest in their integration into scientific workflows. This paper aims to evaluate and compare against human researchers the performance of 6 different LLMs in conducting the various tasks necessary to produce a systematic literature review. The evaluation of the 6 LLMs was split into 3 tasks: literature search, article screening and selection (task 1); data extraction and analysis (task 2); final paper drafting (task 3). Their results were compared with a human-produced systematic review on the same topic, serving as reference standard. The evaluation was repeated on two rounds to evaluate between-version changes and improvements of LLMs over time. Out of the 18 scientific articles to be extracted from the literature for task 1, the best LLM managed to identify 13. Data extraction and analysis for task 2 was only partially accurate and cumbersome. The full papers generated by LLMs for task 3 were short and uninspiring, often not fully adhering to the standard PRISMA 2020 template for a systematic review. Currently, LLMs are not capable of conducting a scientific systematic review in the medical domain without prompt-engineering strategies. However, their capabilities are advancing rapidly, and, with an appropriate supervision they can provide valuable support throughout the review process.

The online version contains supplementary material available at 10.1038/s41598-025-28993-5.