SeBERTis: A Framework for Producing Classifiers of Security-Related Issue Reports

TL;DR

SeBERTis introduces a novel framework using fine-tuned transformer models to detect security-related issue reports in software repositories, significantly outperforming existing methods in accuracy and robustness.

Contribution

The paper presents SEBERTIS, a new approach that trains DNN classifiers on semantic surrogates to improve detection of unseen security issues, overcoming lexical cue reliance of prior models.

Findings

Achieved 0.9880 F1-score on a 10,000-issue dataset.

Outperformed state-of-the-art classifiers by up to 97% in detection metrics.

Surpassed LLM baselines with 23-64% higher precision, recall, and F1-score.

Abstract

Monitoring issue tracker submissions is a crucial software maintenance activity. A key goal is the prioritization of high risk, security-related bugs. If such bugs can be recognized early, the risk of propagation to dependent products and endangerment of stakeholder benefits can be mitigated. To assist triage engineers with this task, several automatic detection techniques, from Machine Learning (ML) models to prompting Large Language Models (LLMs), have been proposed. Although promising to some extent, prior techniques often memorize lexical cues as decision shortcuts, yielding low detection rate specifically for more complex submissions. As such, these classifiers do not yet reach the practical expectations of a real-time detector of security-related issues. To address these limitations, we propose SEBERTIS, a framework to train Deep Neural Networks (DNNs) as classifiers independent of lexical cues, so that they can confidently detect fully unseen security-related issues. SEBERTIS capitalizes on fine-tuning bidirectional transformer architectures as Masked Language Models (MLMs) on a series of semantically equivalent vocabulary to prediction labels (which we call Semantic Surrogates) when they have been replaced with a mask. Our SEBERTIS-trained classifier achieves a 0.9880 F1-score in detecting security-related issues of a curated corpus of 10,000 GitHub issue reports, substantially outperforming state-of-the-art issue classifiers, with 14.44%-96.98%, 15.40%-93.07%, and 14.90%-94.72% higher detection precision, recall, and F1-score over ML-based baselines. Our classifier also substantially surpasses LLM baselines, with an improvement of 23.20%-63.71%, 36.68%-85.63%, and 39.49%-74.53% for precision, recall, and F1-score.