Mapping the Bid Behavior of Conference Referees

TL;DR

This paper investigates how conference referees bid on submissions, revealing that bids are somewhat influenced by subject domain but also affected by other factors, impacting review quality.

Contribution

It provides an analysis of referee bid behavior in digital library conferences, highlighting the partial influence of subject domain on bidding decisions.

Findings

Referees tend to bid based on subject domain.

The relationship between bid and domain is weak.

Other factors influence referee bidding behavior.

Abstract

The peer-review process, in its present form, has been repeatedly criticized. Of the many critiques ranging from publication delays to referee bias, this paper will focus specifically on the issue of how submitted manuscripts are distributed to qualified referees. Unqualified referees, without the proper knowledge of a manuscript's domain, may reject a perfectly valid study or potentially more damaging, unknowingly accept a faulty or fraudulent result. In this paper, referee competence is analyzed with respect to referee bid data collected from the 2005 Joint Conference on Digital Libraries (JCDL). The analysis of the referee bid behavior provides a validation of the intuition that referees are bidding on conference submissions with regards to the subject domain of the submission. Unfortunately, this relationship is not strong and therefore suggests that there exists other factors beyond subject domain that may be influencing referees to bid for particular submissions.