Automated Peer Reviewing in Paper SEA: Standardization, Evaluation, and Analysis

TL;DR

This paper presents SEA, an automated peer review framework using LLMs, which standardizes reviews, generates constructive feedback, and evaluates review consistency, aiming to improve scientific publication quality.

Contribution

The paper introduces a novel three-module framework SEA that standardizes, evaluates, and analyzes peer reviews using LLMs, with a new mismatch score metric and self-correction strategy.

Findings

SEA generates valuable insights for authors.

The mismatch score effectively assesses review consistency.

Experimental results show improved review quality.

Abstract

In recent years, the rapid increase in scientific papers has overwhelmed traditional review mechanisms, resulting in varying quality of publications. Although existing methods have explored the capabilities of Large Language Models (LLMs) for automated scientific reviewing, their generated contents are often generic or partial. To address the issues above, we introduce an automated paper reviewing framework SEA. It comprises of three modules: Standardization, Evaluation, and Analysis, which are represented by models SEA-S, SEA-E, and SEA-A, respectively. Initially, SEA-S distills data standardization capabilities of GPT-4 for integrating multiple reviews for a paper. Then, SEA-E utilizes standardized data for fine-tuning, enabling it to generate constructive reviews. Finally, SEA-A introduces a new evaluation metric called mismatch score to assess the consistency between paper contents and reviews. Moreover, we design a self-correction strategy to enhance the consistency. Extensive experimental results on datasets collected from eight venues show that SEA can generate valuable insights for authors to improve their papers.