Evaluating Diverse Large Language Models for Automatic and General Bug Reproduction

TL;DR

This paper introduces LIBRO, a novel approach using large language models to automatically generate bug-reproducing tests from natural language bug reports, achieving significant success on the Defects4J benchmark and demonstrating the potential of open-source LLMs.

Contribution

The paper presents LIBRO, a new technique leveraging LLMs for automatic bug reproduction from natural language reports, including extensive evaluation across multiple models and benchmarks.

Findings

LIBRO reproduces about one-third of bugs in Defects4J.

Open-source LLMs like StarCoder achieve 70-90% of the performance of closed-source models.

Bug reproduction improves with larger LLM sizes.

Abstract

Bug reproduction is a critical developer activity that is also challenging to automate, as bug reports are often in natural language and thus can be difficult to transform to test cases consistently. As a result, existing techniques mostly focused on crash bugs, which are easier to automatically detect and verify. In this work, we overcome this limitation by using large language models (LLMs), which have been demonstrated to be adept at natural language processing and code generation. By prompting LLMs to generate bug-reproducing tests, and via a post-processing pipeline to automatically identify promising generated tests, our proposed technique LIBRO could successfully reproduce about one-third of all bugs in the widely used Defects4J benchmark. Furthermore, our extensive evaluation on 15 LLMs, including 11 open-source LLMs, suggests that open-source LLMs also demonstrate substantial potential, with the StarCoder LLM achieving 70% of the reproduction performance of the closed-source OpenAI LLM code-davinci-002 on the large Defects4J benchmark, and 90% of performance on a held-out bug dataset likely not part of any LLM's training data. In addition, our experiments on LLMs of different sizes show that bug reproduction using LIBRO improves as LLM size increases, providing information as to which LLMs can be used with the LIBRO pipeline.