ARIES: A Corpus of Scientific Paper Edits Made in Response to Peer Reviews

TL;DR

This paper introduces ARIES, a dataset of scientific paper revisions made in response to peer reviews, and evaluates models' ability to link comments to edits and generate appropriate revisions, revealing current limitations.

Contribution

The paper presents ARIES, a new dataset for studying automated paper revision based on peer feedback, and analyzes model performance and GPT-4's capabilities in this task.

Findings

Models struggle to identify comment-edit relationships, especially with indirect reasoning.

GPT-4 often produces superficial edits that lack technical depth.

High inter-annotator agreement on the expert-labeled test set.

Abstract

We introduce the task of automatically revising scientific papers based on peer feedback and release ARIES, a dataset of review comments and their corresponding paper edits. The data is drawn from real reviewer-author interactions from computer science, and we provide labels linking each reviewer comment to the specific paper edits made by the author in response. We automatically create a high-precision silver training set, as well as an expert-labeled test set that shows high inter-annotator agreement. In experiments with 10 models covering the state of the art, we find that they struggle even to identify which edits correspond to a comment -- especially when the relationship between the edit and the comment is indirect and requires reasoning to uncover. We also extensively analyze GPT-4's ability to generate edits given a comment and the original paper. We find that it often succeeds on a superficial level, but tends to rigidly follow the wording of the feedback rather than the underlying intent, and lacks technical details compared to human-written edits.