Does double-blind peer-review reduce bias? Evidence from a top computer science conference

TL;DR

This study analyzes the impact of switching from single-blind to double-blind peer review at ICLR, finding that double-blind review reduces prestige bias and may improve the quality of accepted papers, with mixed effects on scores and acceptance.

Contribution

It provides empirical evidence that double-blind peer review can reduce prestige bias and improve paper quality assessment in a top computer science conference.

Findings

Scores for prestigious authors decreased after switching to double-blind review.

Double-blind review may better identify lower-quality papers.

Changing rating scales also affected bias and acceptance decisions.

Abstract

Peer review is widely regarded as essential for advancing scientific research. However, reviewers may be biased by authors' prestige or other characteristics. Double-blind peer review, in which the authors' identities are masked from the reviewers, has been proposed as a way to reduce reviewer bias. Although intuitive, evidence for the effectiveness of double-blind peer review in reducing bias is limited and mixed. Here, we examine the effects of double-blind peer review on prestige bias by analyzing the peer review files of 5027 papers submitted to the International Conference on Learning Representations (ICLR), a top computer science conference that changed its reviewing policy from single-blind peer review to double-blind peer review in 2018. We find that after switching to double-blind review, the scores given to the most prestigious authors significantly decreased. However, because many of these papers were above the threshold for acceptance, the change did not affect paper acceptance decisions significantly. Nevertheless, we show that double-blind peer review may have improved the quality of the selections by limiting other (non-author-prestige) biases. Specifically, papers rejected in the single-blind format are cited more than those rejected under the double-blind format, suggesting that double-blind review better identifies poorer quality papers. Interestingly, an apparently unrelated change - the change of rating scale from 10 to 4 points - likely reduced prestige bias significantly, to an extent that affected papers' acceptance. These results provide some support for the effectiveness of double-blind review in reducing prestige bias, while opening new research directions on the impact of peer review formats.