BugMentor: Generating Answers to Follow-up Questions from Software Bug Reports using Structured Information Retrieval and Neural Text Generation

TL;DR

BugMentor is a novel method that combines structured retrieval and neural text generation to produce accurate answers to follow-up questions in bug reports, improving developer communication and bug resolution efficiency.

Contribution

It introduces a new approach integrating structured information retrieval with neural text generation to answer follow-up questions in bug reports, outperforming existing baselines.

Findings

Achieved BLEU score up to 72 and semantic similarity up to 92.

Outperformed four baseline methods with statistical significance.

Developer study showed answers were more accurate, concise, and useful.

Abstract

Software bug reports often lack crucial information (e.g., steps to reproduce), which makes bug resolution challenging. Developers thus ask follow-up questions to capture additional information. However, according to existing evidence, bug reporters often face difficulties answering them, which leads to the premature closing of bug reports without any resolution. Recent studies suggest follow-up questions to support the developers, but answering the follow-up questions still remains a major challenge. In this paper, we propose BugMentor, a novel approach that combines structured information retrieval and neural text generation (e.g., Mistral) to generate appropriate answers to the follow-up questions. Our technique identifies the past relevant bug reports to a given bug report, captures contextual information, and then leverages it to generate the answers. We evaluate our generated answers against the ground truth answers using four appropriate metrics, including the BLEU Score and the Semantic Similarity. We achieve a BLEU Score of up to 72 and a Semantic Similarity of up to 92, indicating that our technique can generate understandable and good answers to the follow-up questions according to Google's AutoML Translation documentation. Our technique also outperforms four existing baselines with a statistically significant margin. We also conduct a developer study involving 23 participants where the answers from our technique were found to be more accurate, more precise, more concise and more useful.