What Can We Do to Improve Peer Review in NLP?

TL;DR

This paper discusses the challenges in NLP peer review, highlighting the need for better incentives and mechanisms to improve review quality and consistency.

Contribution

It identifies key issues in current NLP peer review and proposes directions for creating effective incentives and mechanisms for improvement.

Findings

Current peer review is often inconsistent and unreliable.

Incentive structures are crucial for improving review quality.

Mechanisms for standardizing reviews are needed.

Abstract

Peer review is our best tool for judging the quality of conference submissions, but it is becoming increasingly spurious. We argue that a part of the problem is that the reviewers and area chairs face a poorly defined task forcing apples-to-oranges comparisons. There are several potential ways forward, but the key difficulty is creating the incentives and mechanisms for their consistent implementation in the NLP community.