Towards an OSF-based Registered Report Template for Software Engineering Controlled Experiments
Ana B. M. Bett, Thais S. Nepomuceno, Edson OliveiraJr, Maria Teresa Baldassarre, Valdemar V. Graciano Neto, Marcos Kalinowski

TL;DR
This paper evaluates the use of OSF-based Registered Reports for software engineering experiments, highlighting gaps in current templates and proposing the need for tailored guidelines to improve rigor, reproducibility, and transparency.
Contribution
It analyzes existing OSF RR templates against documentation guidelines and emphasizes the necessity for developing specific RR guidelines for software engineering.
Findings
Current RR templates lack comprehensive coverage of guidelines.
OSF RR templates have limited customization options.
Adopting tailored RR guidelines can improve experiment rigor.
Abstract
Context: The empirical software engineering (ESE) community has contributed to improving experimentation over the years. However, there is still a lack of rigor in describing controlled experiments, hindering reproducibility and transparency. Registered Reports (RR) have been discussed in the ESE community to address these issues. A RR registers a study's hypotheses, methods, and/or analyses before execution, involving peer review and potential acceptance before data collection. This helps mitigate problematic practices such as p-hacking, publication bias, and inappropriate post hoc analysis. Objective: This paper presents initial results toward establishing an RR template for Software Engineering controlled experiments using the Open Science Framework (OSF). Method: We analyzed templates of selected OSF RR types in light of documentation guidelines for controlled experiments. Results:…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsScientific Computing and Data Management · Software Engineering Techniques and Practices · Software Engineering Research
