Large language models for automated scholarly paper review: A survey

TL;DR

This survey reviews the use of large language models in automating scholarly paper review, covering current methods, challenges, datasets, and future directions to advance the field.

Contribution

It provides a comprehensive overview of LLM-based automated scholarly paper review, including technological bottlenecks, new tools, and stakeholder attitudes.

Findings

LLMs are increasingly used in ASPR with promising results.

Technological challenges in ASPR are being addressed with new LLM techniques.

Stakeholder attitudes towards ASPR vary, influencing its adoption.

Abstract

Large language models (LLMs) have significantly impacted human society, influencing various domains. Among them, academia is not simply a domain affected by LLMs, but it is also the pivotal force in the development of LLMs. In academic publication, this phenomenon is represented during the incorporation of LLMs into the peer review mechanism for reviewing manuscripts. LLMs hold transformative potential for the full-scale implementation of automated scholarly paper review (ASPR), but they also pose new issues and challenges that need to be addressed. In this survey paper, we aim to provide a holistic view of ASPR in the era of LLMs. We begin with a survey to find out which LLMs are used to conduct ASPR. Then, we review what ASPR-related technological bottlenecks have been solved with the incorporation of LLM technology. After that, we move on to explore new methods, new datasets, new source code, and new online systems that come with LLMs for ASPR. Furthermore, we summarize the performance and issues of LLMs in ASPR, and investigate the attitudes and reactions of publishers and academia to ASPR. Lastly, we discuss the challenges and future directions associated with the development of LLMs for ASPR. This survey serves as an inspirational reference for the researchers and can promote the progress of ASPR for its actual implementation.