A Comprehensive Study on Large Language Models for Mutation Testing

TL;DR

This study empirically evaluates large language models for mutation testing, showing they generate more effective and diverse mutants than rule-based methods but with higher non-compilability and duplication rates.

Contribution

It provides the first comprehensive empirical comparison of LLMs for mutation testing, establishing a baseline for effectiveness and highlighting trade-offs in mutant quality.

Findings

LLMs produce more diverse mutants closer to real bugs.

LLMs achieve 111.29% higher fault detection than rule-based methods.

Mutants from LLMs have higher non-compilability and duplication rates.

Abstract

Large Language Models (LLMs) have recently been used to generate mutants in both research work and in industrial practice. However, there has been no comprehensive empirical study of their performance for this increasingly important LLM-based Software Engineering application. To address this, we conduct a comprehensive empirical study evaluating BugFarm and LLMorpheus (the two state-of-the-art LLM-based approaches), alongside seven LLMs using our newly designed prompt, including both leading open- and closed-source models, on 851 real bugs from two Java real-world bug benchmarks. Our results reveal that, compared to existing rule-based approaches, LLMs generate more diverse mutants, that are behaviorally closer to real bugs and, most importantly, with 111.29% higher fault detection. That is, 87.98% (for LLMs) vs. 41.64% (for rule-based); an increase of 46.34 percentage points. Nevertheless, our results also reveal that these impressive results for improved effectiveness come at a cost: the LLM-generated mutants have worse non-compilability, duplication, and equivalent mutant rates by 26.60, 10.14, and 3.51 percentage points, respectively. These findings are immediately actionable for both research and practice. They allow practitioners to have greater confidence in deploying LLM-based mutation, while researchers now have a baseline for the state-of-the-art, with which they can research techniques to further improve effectiveness and reduce cost.