Double blind vs. open review: an evolutionary game logit-simulating the behavior of authors and reviewers

TL;DR

This study uses game theory simulations to compare double-blind and open peer review systems, finding no consistent advantage of either and highlighting the influence of external factors on reviewer and author behavior.

Contribution

It introduces an evolutionary game model to analyze peer review incentives, providing a strategic perspective on review system impacts and challenging assumptions about reform effectiveness.

Findings

No reliable difference between review systems in incentivizing effort.

Higher payoffs for good reviewing can reduce author effort under open review.

External utility of effort significantly influences review dynamics.

Abstract

Despite the tremendous successes of science in providing knowledge and technologies, the Replication Crisis has highlighted that scientific institutions have much room for improvement. Peer-review is one target of criticism and suggested reforms. However, despite numerous controversies peer review systems, plus the obvious complexity of the incentives affecting the decisions of authors and reviewers, there is very little systematic and strategic analysis of peer-review systems. In this paper, we begin to address this feature of the peer-review literature by applying the tools of game theory. We use simulations to develop an evolutionary model based around a game played by authors and reviewers, before exploring some of its tendencies. In particular, we examine the relative impact of double-blind peer-review and open review on incentivising reviewer effort under a variety of parameters. We also compare (a) the impact of one review system versus another with (b) other alterations, such as higher costs of reviewing. We find that is no reliable difference between peer-review systems in our model. Furthermore, under some conditions, higher payoffs for good reviewing can lead to less (rather than more) author effort under open review. Finally, compared to the other parameters that we vary, it is the exogenous utility of author effort that makes an important and reliable difference in our model, which raises the possibility that peer-review might not be an important target for institutional reforms.