Effectiveness of Anonymization in Double-Blind Review

TL;DR

This study evaluates the effectiveness of anonymization in double-blind peer review by analyzing reviewer guesses across multiple conferences, finding that most reviews do not reveal author identities and supporting the continued use of double-blind review.

Contribution

The paper provides empirical evidence on anonymization effectiveness and discusses the implications for double-blind review practices in computer science conferences.

Findings

74%-90% of reviews contain no correct author guess

Reviewers with expertise are more likely to attempt guessing

Double-blind review remains beneficial despite anonymization challenges

Abstract

Double-blind review relies on the authors' ability and willingness to effectively anonymize their submissions. We explore anonymization effectiveness at ASE 2016, OOPSLA 2016, and PLDI 2016 by asking reviewers if they can guess author identities. We find that 74%-90% of reviews contain no correct guess and that reviewers who self-identify as experts on a paper's topic are more likely to attempt to guess, but no more likely to guess correctly. We present our findings, summarize the PC chairs' comments about administering double-blind review, discuss the advantages and disadvantages of revealing author identities part of the way through the process, and conclude by advocating for the continued use of double-blind review.