AssertFlip: Reproducing Bugs via Inversion of LLM-Generated Passing Tests

TL;DR

AssertFlip leverages large language models to generate passing tests and then inverts them to produce bug-reproducing tests, significantly improving bug reproduction success rates in software debugging.

Contribution

Introduces AssertFlip, a novel LLM-based method that inverts passing tests to generate bug-reproducing tests, outperforming existing techniques on the SWT-Bench benchmark.

Findings

Achieves 43.6% success rate on SWT-Bench-Verified subset.

Outperforms all known techniques in bug reproduction benchmarks.

Demonstrates the effectiveness of test inversion for bug reproduction.

Abstract

Bug reproduction is critical in the software debugging and repair process, yet the majority of bugs in open-source and industrial settings lack executable tests to reproduce them at the time they are reported, making diagnosis and resolution more difficult and time-consuming. To address this challenge, we introduce AssertFlip, a novel technique for automatically generating Bug Reproducible Tests (BRTs) using large language models (LLMs). Unlike existing methods that attempt direct generation of failing tests, AssertFlip first generates passing tests on the buggy behaviour and then inverts these tests to fail when the bug is present. We hypothesize that LLMs are better at writing passing tests than ones that crash or fail on purpose. Our results show that AssertFlip outperforms all known techniques in the leaderboard of SWT-Bench, a benchmark curated for BRTs. Specifically, AssertFlip achieves a fail-to-pass success rate of 43.6% on the SWT-Bench-Verified subset.