LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing

TL;DR

This paper investigates how large language models can assist NLP researchers by generating and evaluating paper reviews, comparing their quality to human reviews and assessing their ability to identify review deficiencies.

Contribution

It introduces the ReviewCritique dataset and provides a comprehensive analysis of LLMs' effectiveness in generating and critiquing reviews, a novel exploration in this context.

Findings

LLMs can produce reviews comparable to human reviews in quality.

LLMs are effective at identifying deficiencies in reviews.

The study provides insights into LLMs' potential in assisting peer review processes.

Abstract

This work is motivated by two key trends. On one hand, large language models (LLMs) have shown remarkable versatility in various generative tasks such as writing, drawing, and question answering, significantly reducing the time required for many routine tasks. On the other hand, researchers, whose work is not only time-consuming but also highly expertise-demanding, face increasing challenges as they have to spend more time reading, writing, and reviewing papers. This raises the question: how can LLMs potentially assist researchers in alleviating their heavy workload? This study focuses on the topic of LLMs assist NLP Researchers, particularly examining the effectiveness of LLM in assisting paper (meta-)reviewing and its recognizability. To address this, we constructed the ReviewCritique dataset, which includes two types of information: (i) NLP papers (initial submissions rather than camera-ready) with both human-written and LLM-generated reviews, and (ii) each review comes with "deficiency" labels and corresponding explanations for individual segments, annotated by experts. Using ReviewCritique, this study explores two threads of research questions: (i) "LLMs as Reviewers", how do reviews generated by LLMs compare with those written by humans in terms of quality and distinguishability? (ii) "LLMs as Metareviewers", how effectively can LLMs identify potential issues, such as Deficient or unprofessional review segments, within individual paper reviews? To our knowledge, this is the first work to provide such a comprehensive analysis.