Understanding Peer Review of Software Engineering Papers

TL;DR

This study investigates how software engineering reviewers, including award-winners, evaluate papers, identifying key qualities of good reviews and paper features that influence review outcomes through interviews and a survey of 175 reviewers.

Contribution

It provides empirical insights into reviewer behaviors, criteria for good reviews, and features of papers that lead to positive or negative evaluations in software engineering.

Findings

45% of award-winners review 20+ papers annually

88% of reviewers spend over two hours on journal reviews

Clear, supported validation and novelty are key for positive reviews

Abstract

Peer review is a key activity intended to preserve the quality and integrity of scientific publications. However, in practice it is far from perfect. We aim at understanding how reviewers, including those who have won awards for reviewing, perform their reviews of software engineering papers to identify both what makes a good reviewing approach and what makes a good paper. We first conducted a series of in-person interviews with well-respected reviewers in the software engineering field. Then, we used the results of those interviews to develop a questionnaire used in an online survey and sent out to reviewers from well-respected venues covering a number of software engineering disciplines, some of whom had won awards for their reviewing efforts. We analyzed the responses from the interviews and from 175 reviewers who completed the online survey (including both reviewers who had won awards and those who had not). We report on several descriptive results, including: 45% of award-winners are reviewing 20+ conference papers a year, while 28% of non-award winners conduct that many. 88% of reviewers are taking more than two hours on journal reviews. We also report on qualitative results. To write a good review, the important criteria were it should be factual and helpful, ranked above others such as being detailed or kind. The most important features of papers that result in positive reviews are clear and supported validation, an interesting problem, and novelty. Conversely, negative reviews tend to result from papers that have a mismatch between the method and the claims and from those with overly grandiose claims. The main recommendation for authors is to make the contribution of the work very clear in their paper. In addition, reviewers viewed data availability and its consistency as being important.