Preregistering NLP Research

TL;DR

This paper advocates for adopting preregistration and registered reports in NLP research to enhance transparency, reproducibility, and scientific rigor, inspired by practices in medicine and psychology.

Contribution

It introduces the concept of preregistration to NLP, discusses its potential benefits, and proposes specific preregistration questions for various study types.

Findings

Highlights the benefits of preregistration for NLP research.

Proposes a set of preregistration questions for different study designs.

Encourages community discussion on adopting registered reports.

Abstract

Preregistration refers to the practice of specifying what you are going to do, and what you expect to find in your study, before carrying out the study. This practice is increasingly common in medicine and psychology, but is rarely discussed in NLP. This paper discusses preregistration in more detail, explores how NLP researchers could preregister their work, and presents several preregistration questions for different kinds of studies. Finally, we argue in favour of registered reports, which could provide firmer grounds for slow science in NLP research. The goal of this paper is to elicit a discussion in the NLP community, which we hope to synthesise into a general NLP preregistration form in future research.