An Empirical Study on Failed Error Propagation in Java Programs with Real Faults

TL;DR

This study empirically investigates how often error propagation fails in Java programs with real faults, revealing that failure rates are low at unit level but higher at system level, impacting testing strategies.

Contribution

It provides the first comprehensive empirical analysis of failed error propagation in real Java faults across multiple projects, informing testing practices.

Findings

Failed error propagation is negligible at unit testing level.

Failure rates increase significantly at system testing level.

Method postconditions should focus on externally observable state.

Abstract

During testing, developers can place oracles externally or internally with respect to a method. Given a faulty execution state, i.e., one that differs from the expected one, an oracle might be unable to expose the fault if it is placed at a program point with no access to the incorrect program state or where the program state is no longer corrupted. In such a case, the oracle is subject to failed error propagation. We conducted an empirical study to measure failed error propagation on Defects4J, the reference benchmark for Java programs with real faults, considering all 6 projects available (386 real bugs and 459 fixed methods). Our results indicate that the prevalence of failed error propagation is negligible when testing is performed at the unit level. However, when system-level inputs are provided, the prevalence of failed error propagation increases substantially. This indicates that it is enough for method postconditions to predicate only on the externally observable state/data and that intermediate steps should be checked when testing at system level.