From Issues to Insights: RAG-based Explanation Generation from Software Engineering Artifacts

TL;DR

This paper introduces a novel RAG-based method for generating software explanations from issue-tracking data, demonstrating high alignment with human explanations and enhancing software transparency.

Contribution

It is the first to apply RAG techniques to generate explanations from issue-tracking systems, showcasing feasibility and effectiveness.

Findings

Achieved 90% alignment with human explanations

Demonstrated strong faithfulness and instruction adherence

Proved RAG's potential for software explainability

Abstract

The increasing complexity of modern software systems has made understanding their behavior increasingly challenging, driving the need for explainability to improve transparency and user trust. Traditional documentation is often outdated or incomplete, making it difficult to derive accurate, context-specific explanations. Meanwhile, issue-tracking systems capture rich and continuously updated development knowledge, but their potential for explainability remains untapped. With this work, we are the first to apply a Retrieval-Augmented Generation (RAG) approach for generating explanations from issue-tracking data. Our proof-of-concept system is implemented using open-source tools and language models, demonstrating the feasibility of leveraging structured issue data for explanation generation. Evaluating our approach on an exemplary project's set of GitHub issues, we achieve 90% alignment with human-written explanations. Additionally, our system exhibits strong faithfulness and instruction adherence, ensuring reliable and grounded explanations. These findings suggest that RAG-based methods can extend explainability beyond black-box ML models to a broader range of software systems, provided that issue-tracking data is available - making system behavior more accessible and interpretable.