Enriching Automatic Test Case Generation by Extracting Relevant Test Inputs from Bug Reports

TL;DR

This paper presents BRMiner, a novel method combining Large Language Models with traditional techniques to extract relevant test inputs from bug reports, significantly improving automated test case generation effectiveness and bug detection.

Contribution

Introduction of BRMiner, a hybrid approach leveraging LLMs and traditional methods to enhance relevance of inputs for automated test generation.

Findings

BRMiner achieves a 60.03% Relevant Input Rate.

It attains a 31.71% Relevant Input Extraction Accuracy Rate.

BRMiner improves code coverage and detects more bugs.

Abstract

The quality of software is closely tied to the effectiveness of the tests it undergoes. Manual test writing, though crucial for bug detection, is time-consuming, which has driven significant research into automated test case generation. However, current methods often struggle to generate relevant inputs, limiting the effectiveness of the tests produced. To address this, we introduce BRMiner, a novel approach that leverages Large Language Models (LLMs) in combination with traditional techniques to extract relevant inputs from bug reports, thereby enhancing automated test generation tools. In this study, we evaluate BRMiner using the Defects4J benchmark and test generation tools such as EvoSuite and Randoop. Our results demonstrate that BRMiner achieves a Relevant Input Rate (RIR) of 60.03% and a Relevant Input Extraction Accuracy Rate (RIEAR) of 31.71%, significantly outperforming methods that rely on LLMs alone. The integration of BRMiner's input enhances EvoSuite ability to generate more effective test, leading to increased code coverage, with gains observed in branch, instruction, method, and line coverage across multiple projects. Furthermore, BRMiner facilitated the detection of 58 unique bugs, including those that were missed by traditional baseline approaches. Overall, BRMiner's combination of LLM filtering with traditional input extraction techniques significantly improves the relevance and effectiveness of automated test generation, advancing the detection of bugs and enhancing code coverage, thereby contributing to higher-quality software development.