Reviewer2: Optimizing Review Generation Through Prompt Generation

TL;DR

Reviewer2 is a two-stage framework that enhances automated review generation by explicitly modeling review aspects, resulting in more detailed and comprehensive reviews, supported by a large annotated dataset for future research.

Contribution

This paper introduces Reviewer2, a novel two-stage review generation method that explicitly models review aspects, improving detail and coverage over prior approaches.

Findings

Generated reviews are more detailed and diverse.

The dataset includes 27k papers and 99k annotated reviews.

Reviewer2 outperforms existing automated review methods.

Abstract

Recent developments in LLMs offer new opportunities for assisting authors in improving their work. In this paper, we envision a use case where authors can receive LLM-generated reviews that uncover weak points in the current draft. While initial methods for automated review generation already exist, these methods tend to produce reviews that lack detail, and they do not cover the range of opinions that human reviewers produce. To address this shortcoming, we propose an efficient two-stage review generation framework called Reviewer2. Unlike prior work, this approach explicitly models the distribution of possible aspects that the review may address. We show that this leads to more detailed reviews that better cover the range of aspects that human reviewers identify in the draft. As part of the research, we generate a large-scale review dataset of 27k papers and 99k reviews that we annotate with aspect prompts, which we make available as a resource for future research.