A City upon a Hill: Casting Light on a Real Experimental Process

TL;DR

This ethnography study reveals that actual experimental practices in Empirical Software Engineering differ significantly from textbook procedures, highlighting diverse activities, roles, and perspectives that influence reproducibility efforts.

Contribution

It provides a detailed conceptual and process model of real experimental practices in ESE, addressing gaps between theory and practice.

Findings

Experimental processes are more diverse than textbook models.

Different roles and viewpoints shape experimental procedures.

Models align with community practices but vary in detail.

Abstract

Context: The overall scientific community is proposing measures to improve the reproducibility and replicability of experiments. Reproducibility is relatively easy to achieve. However, replicability is considerably more complex in both the sciences and Empirical Software Engineering (ESE). Several strategies, e.g., replication packages and families of experiments, have been proposed to improve replication in ESE, with limited success. We wonder whether the failures are due to some mismatch, i.e., the researchers' needs are not satisfied by the proposed replication procedures. Objectives: Find out how experimental researchers conduct \textit{experiments in practice}. Methods: We carried out an ethnography study within a SE Research Group. Our main activity was to observe/approach the experimental researchers in their day-to-day settings for two years. Their preferred literature and experimental materials were studied. We used individual and group interviews to gain understanding and examine unclear topics in-depth. Results: We have created conceptual and process models that represent how experimentation is really conducted in the Research Group. Models fit the community's procedures and terminology at a high level, but they become particular in their minute details. Conclusion: The actual experimental process differs from textbooks in several points, namely: (1) Number and diversity of activities, (2) existence of different roles, (3) the granularity of the concepts used by the roles, and (4) the viewpoints that different sub-areas or families of experiments have about the overall process.