A Comparative Study of Transformer-based Neural Text Representation Techniques on Bug Triaging

TL;DR

This study evaluates transformer-based neural text representations, especially DeBERTa, for automating bug triaging, demonstrating significant performance improvements and revealing their varied effectiveness across different bug report types.

Contribution

It is one of the first to fine-tune transformer models like DeBERTa for bug triaging, providing comprehensive quantitative and qualitative analysis on multiple datasets.

Findings

DeBERTa outperforms other models in bug developer and component assignment.

Transformer models significantly improve triaging accuracy over prior methods.

Different models excel at different bug report types.

Abstract

Often, the first step in managing bug reports is related to triaging a bug to the appropriate developer who is best suited to understand, localize, and fix the target bug. Additionally, assigning a given bug to a particular part of a software project can help to expedite the fixing process. However, despite the importance of these activities, they are quite challenging, where days can be spent on the manual triaging process. Past studies have attempted to leverage the limited textual data of bug reports to train text classification models that automate this process -- to varying degrees of success. However, the textual representations and machine learning models used in prior work are limited by their expressiveness, often failing to capture nuanced textual patterns that might otherwise aid in the triaging process. Recently, large, transformer-based, pre-trained neural text representation techniques such as BERT have achieved greater performance in several natural language processing tasks. However, the potential for using these techniques to improve upon prior approaches for automated bug triaging is not well studied or understood. Therefore, in this paper we offer one of the first investigations that fine-tunes transformer-based language models for the task of bug triaging on four open source datasets, spanning a collective 53 years of development history with over 400 developers and over 150 software project components. Our study includes both a quantitative and qualitative analysis of effectiveness. Our findings illustrate that DeBERTa is the most effective technique across the triaging tasks of developer and component assignment, and the measured performance delta is statistically significant compared to other techniques. However, through our qualitative analysis, we also observe that each technique possesses unique abilities best suited to certain types of bug reports.