Exceptional Behaviors: How Frequently Are They Tested?
Andre Hora, Gordon Fraser

TL;DR
This empirical study investigates how often exceptional behaviors are tested in real-world Python systems, revealing that many exceptions are infrequently exercised and highlighting the need for better testing tools and practices.
Contribution
The paper provides the first large-scale empirical analysis of both propagated and non-propagated exceptions in real-world systems, offering insights into testing practices.
Findings
21.4% of methods raise exceptions at runtime
On median, 1 in 10 calls exercise exceptional behaviors
80% of methods that raise exceptions do so infrequently
Abstract
Exceptions allow developers to handle error cases expected to occur infrequently. Ideally, good test suites should test both normal and exceptional behaviors to catch more bugs and avoid regressions. While current research analyzes exceptions that propagate to tests, it does not explore other exceptions that do not reach the tests. In this paper, we provide an empirical study to explore how frequently exceptional behaviors are tested in real-world systems. We consider both exceptions that propagate to tests and the ones that do not reach the tests. For this purpose, we run an instrumented version of test suites, monitor their execution, and collect information about the exceptions raised at runtime. We analyze the test suites of 25 Python systems, covering 5,372 executed methods, 17.9M calls, and 1.4M raised exceptions. We find that 21.4% of the executed methods do raise exceptions at…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsSoftware Testing and Debugging Techniques · Software Engineering Research · Software System Performance and Reliability
