A Model-Based Approach for Specifying Changes in Replications of Empirical Studies in Computer Science

TL;DR

This paper presents a systematic, model-based approach with a supporting tool for reporting changes in empirical study replications, aiming to enhance understanding of validity evolution across research families.

Contribution

It introduces a metamodel, reporting templates, and a software tool for systematically specifying replication changes, validated through multiple case studies in CS and other fields.

Findings

All 92 replication changes were properly specified after improvements.

The approach helps visualize the effects of changes on experimental validity.

The method is applicable beyond Computer Science to other research areas.

Abstract

Context: The need of replicating empirical studies in Computer Science (CS) is widely recognized among the research community to consolidate acquired knowledge generalizing results. It is essential to report the changes of each replication to understand the evolution of the experimental validity across a family of studies. Unfortunately, the lack of proposals undermines these objectives. Objective. The main goal of our work is to provide researchers in CS, and in other areas of research, with a systematic, tool-supported approach for the reporting of changes in the replications of their empirical studies. Method: Applying DSR, we have developed and validated a composite artifact consisting of (i) a metamodel of the relevant concepts of replications and their changes; (ii) templates and linguistic patterns for reporting those concepts; and (iii) a proof-of-concept model-based software tool that supports the proposed approach. For its validation, we have carried out a multiple case study including 9 families of empirical studies from CS and Agrobiology. The 9 families encompass 23 replication studies and 92 replication changes, for which we have analyzed the suitability of our proposal. Results: The multiple case study revealed some initial limitations of our approach related to threats to experimental validity or context variables. After several improvement iterations, all the 92 replication changes could be properly specified, including also their qualitatively estimated effects on experimental validity across the family of experiments and its corresponding visualization. Conclusions: Our proposal for the specification of replication changes seems to fit the needs not only of replications in CS, but also in other research areas. Nevertheless, further research is needed to improve it and to disseminate its use among the research community.