LLMorpheus: Mutation Testing using Large Language Models

TL;DR

This paper introduces LLMorpheus, a mutation testing approach using large language models to generate realistic mutants in JavaScript, addressing limitations of traditional fixed-operator methods.

Contribution

The paper presents a novel mutation testing technique leveraging LLMs to generate more realistic mutants, implemented in a new tool called LLMorpheus for JavaScript.

Findings

LLMorpheus produces mutants resembling real-world bugs.

It outperforms traditional mutation tools like StrykerJS in generating diverse mutants.

The approach is practical in terms of runtime and cost.

Abstract

In mutation testing, the quality of a test suite is evaluated by introducing faults into a program and determining whether the program's tests detect them. Most existing approaches for mutation testing involve the application of a fixed set of mutation operators, e.g., replacing a "+" with a "-", or removing a function's body. However, certain types of real-world bugs cannot easily be simulated by such approaches, limiting their effectiveness. This paper presents a technique for mutation testing where placeholders are introduced at designated locations in a program's source code and where a Large Language Model (LLM) is prompted to ask what they could be replaced with. The technique is implemented in LLMorpheus, a mutation testing tool for JavaScript, and evaluated on 13 subject packages, considering several variations on the prompting strategy, and using several LLMs. We find LLMorpheus to be capable of producing mutants that resemble existing bugs that cannot be produced by StrykerJS, a state-of-the-art mutation testing tool. Moreover, we report on the running time, cost, and number of mutants produced by LLMorpheus, demonstrating its practicality.