A Tool for Generating Exceptional Behavior Tests With Large Language Models

TL;DR

exLong is a framework that uses large language models to automatically generate exceptional behavior tests, helping developers cover error handling scenarios often neglected in traditional testing.

Contribution

This paper introduces exLong, a novel LLM-based framework that automatically generates exceptional behavior tests by reasoning about exception traces and conditional guards.

Findings

Successfully generates comprehensive EBTs for various projects.

Addresses the gap in testing exceptional scenarios compared to happy paths.

Demonstrates effectiveness through a supporting video presentation.

Abstract

Exceptional behavior tests (EBTs) are crucial in software development for verifying that code correctly handles unwanted events and throws appropriate exceptions. However, prior research has shown that developers often prioritize testing "happy paths", e.g., paths without unwanted events over exceptional scenarios. We present exLong, a framework that automatically generates EBTs to address this gap. exLong leverages a large language model (LLM) fine-tuned from CodeLlama and incorporates reasoning about exception-throwing traces, conditional expressions that guard throw statements, and non-exceptional behavior tests that execute similar traces. Our demonstration video illustrates how exLong can effectively assist developers in creating comprehensive EBTs for their project (available at https://youtu.be/Jro8kMgplZk).