Test Wars: A Comparative Study of SBST, Symbolic Execution, and LLM-Based Approaches to Unit Test Generation

TL;DR

This study compares traditional test generation methods (SBST and symbolic execution) with LLM-based approaches, revealing that LLMs excel in semantic understanding and mutation scores but lag in coverage and fault detection.

Contribution

It provides an extensive empirical comparison of three test generation techniques, highlighting the strengths and weaknesses of LLM-based methods in software testing.

Findings

LLMs outperform in mutation scores, indicating deeper semantic understanding.

Traditional methods achieve higher coverage and fault detection.

All tools are affected by class complexity and dependencies, with LLMs being most sensitive.

Abstract

Generating tests automatically is a key and ongoing area of focus in software engineering research. The emergence of Large Language Models (LLMs) has opened up new opportunities, given their ability to perform a wide spectrum of tasks. However, the effectiveness of LLM-based approaches compared to traditional techniques such as search-based software testing (SBST) and symbolic execution remains uncertain. In this paper, we perform an extensive study of automatic test generation approaches based on three tools: EvoSuite for SBST, Kex for symbolic execution, and TestSpark for LLM-based test generation. We evaluate tools performance on the GitBug Java dataset and compare them using various execution-based and feature-based metrics. Our results show that while LLM-based test generation is promising, it falls behind traditional methods in terms of coverage. However, it significantly outperforms them in mutation scores, suggesting that LLMs provide a deeper semantic understanding of code. LLM-based approach also performed worse than SBST and symbolic execution-based approaches w.r.t. fault detection capabilities. Additionally, our feature-based analysis shows that all tools are primarily affected by the complexity and internal dependencies of the class under test (CUT), with LLM-based approaches being especially sensitive to the CUT size.