Replications, Revisions, and Reanalyses: Managing Variance Theories in Software Engineering

TL;DR

This paper proposes a new framework for synthesizing research in software engineering that extends beyond traditional meta-analysis, enabling systematic development of variance theories and better integration of new evidence.

Contribution

It introduces a conceptual framework for evolving variance theories in software engineering, facilitating systematic evidence integration beyond meta-analysis.

Findings

Framework effectively relates new evidence to existing knowledge

Enables systematic expansion of variance theories

Applied to an active research field in SE

Abstract

Variance theories quantify the variance that one or more independent variables cause in a dependent variable. In software engineering (SE), variance theories are used to quantify -- among others -- the impact of tools, techniques, and other treatments on software development outcomes. To acquire variance theories, evidence from individual empirical studies needs to be synthesized to more generally valid conclusions. However, research synthesis in SE is mostly limited to meta-analysis, which requires homogeneity of the synthesized studies to infer generalizable variance. In this paper, we aim to extend the practice of research synthesis beyond meta-analysis. To this end, we derive a conceptual framework for the evolution of variance theories and demonstrate its use by applying it to an active research field in SE. The resulting framework allows researchers to put new evidence in a clear relation to an existing body of knowledge and systematically expand the scientific frontier of a studied phenomenon.