Assessing Exception Handling Testing Practices in Open-Source Libraries

TL;DR

This study evaluates how well open-source Java libraries test exception handling code, revealing coverage gaps but generally effective fault detection capabilities in existing test suites.

Contribution

It provides an empirical analysis of exception handling testing practices and effectiveness in open-source Java libraries, highlighting coverage issues and fault detection performance.

Findings

EH code coverage is lower in catch blocks and throw instructions.

Most libraries detect over 70% of injected faults.

Test suites detected 68% of mutants on average.

Abstract

Modern programming languages (e.g., Java and C#) provide features to separate error-handling code from regular code, seeking to enhance software comprehensibility and maintainability. Nevertheless, the way exception handling (EH) code is structured in such languages may lead to multiple, different, and complex control flows, which may affect the software testability. Previous studies have reported that EH code is typically neglected, not well tested, and its misuse can lead to reliability degradation and catastrophic failures. However, little is known about the relationship between testing practices and EH testing effectiveness. In this exploratory study, we (i) measured the adequacy degree of EH testing concerning code coverage (instruction, branch, and method) criteria; and (ii) evaluated the effectiveness of the EH testing by measuring its capability to detect artificially injected faults (i.e., mutants) using 7 EH mutation operators. Our study was performed using test suites of 27 long-lived Java libraries from open-source ecosystems. Our results show that instructions and branches within $\mathtt{catch}$ blocks and $\mathtt{throw}$ instructions are less covered, with statistical significance than the overall instructions and branches. Nevertheless, most of the studied libraries presented test suites capable of detecting more than 70% of the injected faults. From a total of 12,331 mutants created in this study, the test suites were able to detect 68% of them.