Mitigating Omitted Variable Bias in Empirical Software Engineering

TL;DR

This paper discusses the problem of omitted variable bias in empirical software engineering, illustrating its impact and presenting methods to detect, estimate, and mitigate it using causal models.

Contribution

It introduces analysis techniques based on causal structural models to assess and reduce omitted variable bias in empirical studies in software engineering.

Findings

Omitted variable bias can significantly distort effect estimates in software engineering studies.

Using causal models helps identify and mitigate bias before conducting empirical research.

Investing in bias investigation improves study validity and design.

Abstract

Omitted variable bias occurs when a statistical model leaves out variables that are relevant determinants of the effects under study. This results in the model attributing the missing variables' effect to some of the included variables -- hence over- or under-estimating the latter's true effect. Omitted variable bias presents a significant threat to the validity of empirical research, particularly in non-experimental studies such as those prevalent in empirical software engineering. This paper illustrates the impact of omitted variable bias on two illustrative examples in the software engineering domain, and uses them to present methods to investigate the possible presence of omitted variable bias, to estimate its impact, and to mitigate its drawbacks. The analysis techniques we present are based on causal structural models of the variables of interest, which provide a practical, intuitive summary of the key relations among variables. This paper demonstrates a sequence of analysis steps that inform the design and execution of any empirical study in software engineering. An important observation is that it pays off to invest effort investigating omitted variable bias before actually executing an empirical study, because this effort can lead to a more solid study design, and to a significant reduction in its threats to validity.