Replicability and Reproducibility of a Schema Evolution Study in Embedded Databases

TL;DR

This paper investigates the challenges and efforts involved in replicating and reproducing a schema evolution study in embedded databases, highlighting the importance of detailed reporting for scientific reproducibility.

Contribution

It provides a detailed case study on replicability in database research, emphasizing the need for meticulous documentation and recognition of reproducibility efforts.

Findings

Exact replication achieved for one application

Close results obtained for two applications

Highlights substantial effort needed for reproducibility

Abstract

Ascertaining the feasibility of independent falsification or repetition of published results is vital to the scientific process, and replication or reproduction experiments are routinely performed in many disciplines. Unfortunately, such studies are only scarcely available in database research, with few papers dedicated to re-evaluating published results. In this paper, we conduct a case study on replicating and reproducing a study on schema evolution in embedded databases. We obtain exact results for one out of four database applications studied, and come close in two further cases. By reporting results, efforts, and obstacles encountered, we hope to increase appreciation for the substantial efforts required to ensure reproducibility. By discussing minutiae details required for reproducible work, we argue that such important, but often ignored components of scientific work should receive more credit in the evaluation of future research.