Analyzing Families of Experiments in SE: A Systematic Mapping Study

TL;DR

This study systematically reviews how families of experiments are analyzed in Software Engineering, highlighting the prevalent use of unsuitable techniques and recommending more reliable methods like AD and IPD stratified for better insights.

Contribution

It identifies current aggregation techniques in SE, compares them with mature disciplines, and provides recommendations to improve analysis reliability and transparency.

Findings

Multiple aggregation techniques are used in SE families.

Unsuitable techniques are often chosen according to literature.

AD and IPD stratified are recommended for better analysis.

Abstract

Context: Families of experiments (i.e., groups of experiments with the same goal) are on the rise in Software Engineering (SE). Selecting unsuitable aggregation techniques to analyze families may undermine their potential to provide in-depth insights from experiments' results. Objectives: Identifying the techniques used to aggregate experiments' results within families in SE. Raising awareness of the importance of applying suitable aggregation techniques to reach reliable conclusions within families. Method: We conduct a systematic mapping study (SMS) to identify the aggregation techniques used to analyze families of experiments in SE. We outline the advantages and disadvantages of each aggregation technique according to mature experimental disciplines such as medicine and pharmacology. We provide preliminary recommendations to analyze and report families of experiments in view of families' common limitations with regard to joint data analysis. Results: Several aggregation techniques have been used to analyze SE families of experiments, including Narrative synthesis, Aggregated Data (AD), Individual Participant Data (IPD) mega-trial or stratified, and Aggregation of p-values. The rationale used to select aggregation techniques is rarely discussed within families. Families of experiments are commonly analyzed with unsuitable aggregation techniques according to the literature of mature experimental disciplines. Conclusion: Data analysis' reporting practices should be improved to increase the reliability and transparency of joint results. AD and IPD stratified appear to be suitable to analyze SE families of experiments.