How Testable is Business Software?

TL;DR

This paper analyzes the unit-testability of business-critical Java software, highlighting the challenges in testing due to code design and the implications for automated testing tools and test coverage metrics.

Contribution

It provides an extensive empirical analysis of Java software's testability, revealing the distribution of trivial, testable, and non-testable code segments.

Findings

Most code is either trivial, testable, or hard to test.

Poor testability hampers automated verification and test generation.

Test coverage metrics may be misleading without considering testability.

Abstract

Most businesses rely on a significant stack of software to perform their daily operations. This software is business-critical as defects in this software have major impacts on revenue and customer satisfaction. The primary means for verification of this software is testing. We conducted an extensive analysis of Java software packages to evaluate their unit-testability. The results show that code in software repositories is typically split into portions of very trivial code, non-trivial code that is unit-testable, and code that cannot be unit-tested easily. This brings up interesting considerations regarding the use of test coverage metrics and design for testability, which is crucial for testing efficiency and effectiveness. Lack of unit-testability is an obstacle to applying tools that perform automated verification and test generation. These tools cannot make up for poor testability of the code and have a hard time in succeeding or are not even applicable without first improving the design of the software system.