Registered Reports in Software Engineering

TL;DR

Registered reports in software engineering involve peer-reviewed research protocols that promote transparency, reduce questionable practices, and facilitate early feedback, thereby improving research quality and reproducibility.

Contribution

This paper explains the motivation, implementation, and challenges of adopting registered reports in software engineering research.

Findings

Registered reports have been adopted in multiple conferences and journals.

They help prevent questionable research practices.

Early feedback improves research design quality.

Abstract

Registered reports are scientific publications which begin the publication process by first having the detailed research protocol, including key research questions, reviewed and approved by peers. Subsequent analysis and results are published with minimal additional review, even if there was no clear support for the underlying hypothesis, as long as the approved protocol is followed. Registered reports can prevent several questionable research practices and give early feedback on research designs. In software engineering research, registered reports were first introduced in the International Conference on Mining Software Repositories (MSR) in 2020. They are now established in three conferences and two pre-eminent journals, including Empirical Software Engineering. We explain the motivation for registered reports, outline the way they have been implemented in software engineering, and outline some ongoing challenges for addressing high quality software engineering research.