Comparing Techniques for Aggregating Interrelated Replications in Software Engineering

TL;DR

This paper compares different techniques for aggregating results from interrelated replications in Software Engineering, emphasizing the importance of using raw data for reliable joint conclusions and recommending combined AD and IPD meta-analysis.

Contribution

It provides a systematic comparison of aggregation techniques in SE, highlighting the advantages of AD and IPD meta-analyses over simpler methods.

Findings

Narrative synthesis and p-value aggregation lack full data utilization.

AD meta-analysis offers visual summaries and moderator assessment.

IPD meta-analysis enables interpretation in natural units and detailed moderator analysis.

Abstract

Context: Researchers from different groups and institutions are collaborating towards the construction of groups of interrelated replications. Applying unsuitable techniques to aggregate interrelated replications' results may impact the reliability of joint conclusions. Objectives: Comparing the advantages and disadvantages of the techniques applied to aggregate interrelated replications' results in Software Engineering (SE). Method: We conducted a literature review to identify the techniques applied to aggregate interrelated replications' results in SE. We analyze a prototypical group of interrelated replications in SE with the techniques that we identified. We check whether the advantages and disadvantages of each technique -according to mature experimental disciplines such as medicine- materialize in the SE context. Results: Narrative synthesis and Aggregation of p-values do not take advantage of all the information contained within the raw-data for providing joint conclusions. Aggregated Data (AD) meta-analysis provides visual summaries of results and allows assessing experiment-level moderators. Individual Participant Data (IPD) meta-analysis allows interpreting results in natural units and assessing experiment-level and participant-level moderators. Conclusion: All the information contained within the raw-data should be used to provide joint conclusions. AD and IPD, when used in tandem, seem suitable to analyze groups of interrelated replications in SE.