Towards an OSF-based Registered Report Template for Software Engineering Controlled Experiments

TL;DR

This paper evaluates the use of OSF-based Registered Reports for software engineering experiments, highlighting gaps in current templates and proposing the need for tailored guidelines to improve rigor, reproducibility, and transparency.

Contribution

It analyzes existing OSF RR templates against documentation guidelines and emphasizes the necessity for developing specific RR guidelines for software engineering.

Findings

Current RR templates lack comprehensive coverage of guidelines.

OSF RR templates have limited customization options.

Adopting tailored RR guidelines can improve experiment rigor.

Abstract

Context: The empirical software engineering (ESE) community has contributed to improving experimentation over the years. However, there is still a lack of rigor in describing controlled experiments, hindering reproducibility and transparency. Registered Reports (RR) have been discussed in the ESE community to address these issues. A RR registers a study's hypotheses, methods, and/or analyses before execution, involving peer review and potential acceptance before data collection. This helps mitigate problematic practices such as p-hacking, publication bias, and inappropriate post hoc analysis. Objective: This paper presents initial results toward establishing an RR template for Software Engineering controlled experiments using the Open Science Framework (OSF). Method: We analyzed templates of selected OSF RR types in light of documentation guidelines for controlled experiments. Results: The observed lack of rigor motivated our investigation of OSF-based RR types. Our analysis showed that, although one of the RR types aligned with many of the documentation suggestions contained in the guidelines, none of them covered the guidelines comprehensively. The study also highlights limitations in OSF RR template customization. Conclusion: Despite progress in ESE, planning and documenting experiments still lack rigor, compromising reproducibility. Adopting OSF-based RRs is proposed. However, no currently available RR type fully satisfies the guidelines. Establishing RR-specific guidelines for SE is deemed essential.