Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based Interactions

TL;DR

This paper redefines peer review as a multi-turn, role-based dialogue using a large dataset, aiming to improve LLM applications by capturing the iterative and interactive nature of real-world peer reviews.

Contribution

It introduces a comprehensive dataset and evaluation metrics for modeling peer review as a multi-turn dialogue with distinct roles, advancing LLM-based peer review systems.

Findings

Constructed a dataset with over 26,841 papers and 92,017 reviews.

Proposed metrics for evaluating role-based LLM performance in peer review.

Facilitates dynamic, interactive peer review simulation using LLMs.

Abstract

Large Language Models (LLMs) have demonstrated wide-ranging applications across various fields and have shown significant potential in the academic peer-review process. However, existing applications are primarily limited to static review generation based on submitted papers, which fail to capture the dynamic and iterative nature of real-world peer reviews. In this paper, we reformulate the peer-review process as a multi-turn, long-context dialogue, incorporating distinct roles for authors, reviewers, and decision makers. We construct a comprehensive dataset containing over 26,841 papers with 92,017 reviews collected from multiple sources, including the top-tier conference and prestigious journal. This dataset is meticulously designed to facilitate the applications of LLMs for multi-turn dialogues, effectively simulating the complete peer-review process. Furthermore, we propose a series of metrics to evaluate the performance of LLMs for each role under this reformulated peer-review setting, ensuring fair and comprehensive evaluations. We believe this work provides a promising perspective on enhancing the LLM-driven peer-review process by incorporating dynamic, role-based interactions. It aligns closely with the iterative and interactive nature of real-world academic peer review, offering a robust foundation for future research and development in this area. We open-source the dataset at https://github.com/chengtan9907/ReviewMT.