Improving Software Engineering Research through Experimentation Workbenches

TL;DR

This paper advocates for the development of experimentation workbenches to enhance reproducibility, repeatability, and support for experimentation activities in software engineering research.

Contribution

It introduces the concept of experimentation workbenches, outlining core requirements and benefits, and illustrates their application through a scenario and case study.

Findings

Experimentation workbenches improve reproducibility of research.

They facilitate easier experimentation and replication.

Case studies demonstrate practical benefits.

Abstract

Experimentation with software prototypes plays a fundamental role in software engineering research. In contrast to many other scientific disciplines, however, explicit support for this key activity in software engineering is relatively small. While some approaches to improve this situation have been proposed by the software engineering community, experiments are still very difficult and sometimes impossible to replicate. In this paper, we propose the concept of an experimentation workbench as a means of explicit support for experimentation in software engineering research. In particular, we discuss core requirements that an experimentation workbench should satisfy in order to qualify as such and to offer a real benefit for researchers. Beyond their core benefits for experimentation, we stipulate that experimentation workbenches will also have benefits in regard to reproducibility and repeatability of software engineering research. Further, we illustrate this concept with a scenario and a case study, and describe relevant challenges as well as our experience with experimentation workbenches.