Large language models for post-publication research evaluation: Evidence from expert recommendations and citation indicators

TL;DR

This paper investigates the potential of large language models to automate post-publication research evaluation by benchmarking their performance against expert judgments and citation metrics, highlighting strengths and limitations.

Contribution

It systematically evaluates various LLMs on tasks like identifying high-quality articles and detailed rating, demonstrating the impact of different prompting and training strategies.

Findings

LLMs achieve over 0.8 accuracy in coarse evaluation tasks.

Performance drops significantly in fine-grained rating tasks.

Supervised fine-tuning yields the best balanced results.

Abstract

Assessing the quality of scientific research is essential for scholarly communication, yet widely used approaches face limitations in scalability, subjectivity, and time delay. Recent advances in large language models (LLMs) offer new opportunities for automated research evaluation based on textual content. This study examines whether LLMs can support post-publication peer review tasks by benchmarking their outputs against expert judgments and citation-based indicators. Two evaluation tasks are constructed using articles from the H1 Connect platform: identifying high-quality articles and performing finer-grained evaluation including article rating, merit classification, and expert style commenting. Multiple model families, including BERT models, general-purpose LLMs, and reasoning oriented LLMs, are evaluated under multiple learning strategies. Results show that LLMs perform well in coarse grained evaluation tasks, achieving accuracy above 0.8 in identifying highly recommended articles. However, performance decreases substantially in fine-grained rating tasks. Few-shot prompting improves performance over zero-shot settings, while supervised fine-tuning produces the strongest and most balanced results. Retrieval augmented prompting improves classification accuracy in some cases but does not consistently strengthen alignment with citation indicators. The overall correlations between model outputs and citation indicators remain positive but moderate.