Causal Inference for the Effect of Code Coverage on Bug Introduction

TL;DR

This study investigates the causal impact of code coverage levels on bug introduction in open source JavaScript and TypeScript projects, using causal inference methods to move beyond correlational analyses.

Contribution

It introduces a causal inference framework with a directed acyclic graph and propensity score adjustment to quantify the effect of code coverage on bugs, accounting for confounders.

Findings

Quantifies the causal effect of code coverage on bug introduction.

Identifies potential non-linear dose-response relationships.

Provides insights into optimal coverage levels for software quality.

Abstract

Context: Code coverage is widely used as a software quality assurance measure. However, its effect, and specifically the advisable dose, are disputed in both the research and engineering communities. Prior work reports only correlational associations, leaving results vulnerable to confounding factors. Objective: We aim to quantify the causal effect of code coverage (exposure) on bug introduction (outcome) in the context of mature JavaScript and TypeScript open source projects, addressing both the overall effect and its variance across coverage levels. Method: We construct a causal directed acyclic graph to identify confounders within the software engineering process, modeling key variables from the source code, issue- and review systems, and continuous integration. Using generalized propensity score adjustment, we will apply doubly robust regression-based causal inference for continuous exposure to a novel dataset of bug-introducing and non-bug-introducing changes. We estimate the average treatment effect and dose-response relationship to examine potential non-linear patterns (e.g., thresholds or diminishing returns) within the projects of our dataset.