Mokav: Execution-driven Differential Testing with LLMs

TL;DR

Mokav is an execution-driven tool leveraging large language models to generate difference exposing tests between program versions, significantly improving detection success rates in software testing tasks.

Contribution

The paper introduces Mokav, a novel LLM-based, execution-driven approach for generating tests that reveal functional differences between program versions, outperforming existing methods.

Findings

Mokav detects differences in 81.7% of program pairs.

It outperforms Pynguin and Differential Prompting significantly.

Iterative, execution-driven feedback enhances detection effectiveness.

Abstract

It is essential to detect functional differences between programs in various software engineering tasks, such as automated program repair, mutation testing, and code refactoring. The problem of detecting functional differences between two programs can be reduced to searching for a difference exposing test (DET): a test input that results in different outputs on the subject programs. In this paper, we propose Mokav, a novel execution-driven tool that leverages LLMs to generate DETs. Mokav takes two versions of a program (P and Q) and an example test input. When successful, Mokav generates a valid DET, a test input that leads to provably different outputs on P and Q. Mokav iteratively prompts an LLM with a specialized prompt to generate new test inputs. At each iteration, Mokav provides execution-based feedback from previously generated tests until the LLM produces a DET. We evaluate Mokav on 1535 pairs of Python programs collected from the Codeforces competition platform and 32 pairs of programs from the QuixBugs dataset. Our experiments show that Mokav outperforms the state-of-the-art, Pynguin and Differential Prompting, by a large margin. Mokav can generate DETs for 81.7% (1,255/1535) of the program pairs in our benchmark (versus 4.9% for Pynguin and 37.3% for Differential Prompting). We demonstrate that the iterative and execution-driven feedback components of the system contribute to its high effectiveness.