Automatic Analysis of Substantiation in Scientific Peer Reviews

TL;DR

This paper introduces an automatic method to evaluate the substantiation quality in scientific peer reviews by creating a new dataset and training an argument mining system, aiming to improve peer review quality control.

Contribution

The paper presents the first annotated dataset for claim-evidence extraction in peer reviews and develops an argument mining system to assess review substantiation automatically.

Findings

The dataset contains 550 reviews from NLP conferences.

The argument mining system effectively analyzes substantiation levels.

Insights into peer reviewing quality trends in NLP conferences.

Abstract

With the increasing amount of problematic peer reviews in top AI conferences, the community is urgently in need of automatic quality control measures. In this paper, we restrict our attention to substantiation -- one popular quality aspect indicating whether the claims in a review are sufficiently supported by evidence -- and provide a solution automatizing this evaluation process. To achieve this goal, we first formulate the problem as claim-evidence pair extraction in scientific peer reviews, and collect SubstanReview, the first annotated dataset for this task. SubstanReview consists of 550 reviews from NLP conferences annotated by domain experts. On the basis of this dataset, we train an argument mining system to automatically analyze the level of substantiation in peer reviews. We also perform data analysis on the SubstanReview dataset to obtain meaningful insights on peer reviewing quality in NLP conferences over recent years.