exLong: Generating Exceptional Behavior Tests with Large Language Models

TL;DR

exLong is a novel framework using large language models to automatically generate exceptional behavior tests, addressing the common developer focus on happy paths and improving test coverage for exception handling.

Contribution

This paper introduces exLong, the first LLM-based framework specifically designed to generate exceptional behavior tests for programming languages.

Findings

exLong outperforms state-of-the-art models like CAT-LM and GPT-4o.

23 generated EBTs by exLong were accepted into open-source projects.

exLong effectively captures reasoning about exception traces and guards.

Abstract

Many popular programming languages, including C#, Java, and Python, support exceptions. Exceptions are thrown during program execution if an unwanted event happens, e.g., a method is invoked with an illegal argument value. Software developers write exceptional behavior tests (EBTs) to check that their code detects unwanted events and throws appropriate exceptions. Prior research studies have shown the importance of EBTs, but those studies also highlighted that developers put most of their efforts on "happy paths", e.g., paths without unwanted events. To help developers fill the gap, we present the first framework, dubbed exLong, that automatically generates EBTs. exLong is a large language model instruction fine-tuned from CodeLlama and embeds reasoning about traces that lead to throw statements, conditional expressions that guard throw statements, and non-exceptional behavior tests that execute similar traces. We compare exLong with the state-of-the-art models for test generation (CAT-LM) and one of the strongest foundation models (GPT-4o), as well as with analysis-based tools for test generation (Randoop and EvoSuite). Our results show that exLong outperforms existing models and tools. Furthermore, we contributed several pull requests to open-source projects and 23 EBTs generated by exLong were already accepted.