Why are Some Bugs Non-Reproducible? An Empirical Investigation using Data Fusion

TL;DR

This study investigates why some software bugs are non-reproducible by analyzing bug reports and developer behaviors, offering insights and tools to improve bug reproducibility and software reliability.

Contribution

It provides the first comprehensive empirical analysis of factors leading to non-reproducible bugs and explores developer strategies for handling such bugs.

Findings

Identified 11 key factors influencing bug non-reproducibility

Developers often close or seek more info on non-reproducible bugs

Proposed actionable solutions like false-positive detectors and sandboxes

Abstract

Software developers attempt to reproduce software bugs to understand their erroneous behaviours and to fix them. Unfortunately, they often fail to reproduce (or fix) them, which leads to faulty, unreliable software systems. However, to date, only a little research has been done to better understand what makes the software bugs non-reproducible. In this paper, we conduct a multimodal study to better understand the non-reproducibility of software bugs. First, we perform an empirical study using 576 non-reproducible bug reports from two popular software systems (Firefox, Eclipse) and identify 11 key factors that might lead a reported bug to non-reproducibility. Second, we conduct a user study involving 13 professional developers where we investigate how the developers cope with non-reproducible bugs. We found that they either close these bugs or solicit for further information, which involves long deliberations and counter-productive manual searches. Third, we offer several actionable insights on how to avoid non-reproducibility (e.g., false-positive bug report detector) and improve reproducibility of the reported bugs (e.g., sandbox for bug reproduction) by combining our analyses from multiple studies (e.g., empirical study, developer study).