How to Better Distinguish Security Bug Reports (using Dual Hyperparameter Optimization

TL;DR

This paper introduces Swift, a dual hyperparameter optimizer that significantly improves the recall of security bug report classification by jointly optimizing data preprocessing and learning algorithms, outperforming previous methods.

Contribution

The paper presents Swift, a novel dual optimizer that effectively enhances security bug report detection by optimizing both pre-processing and learning parameters simultaneously.

Findings

Swift achieves median recall of 77.4% on Chromium data, compared to 15.7% by FARSEC.

In Ambari data, Swift's median recall improves from 21.5% to 85.7%.

Recall improvements come with moderate false positive rate increases from 8% to 24%.

Abstract

Background: In order that the general public is not vulnerable to hackers, security bug reports need to be handled by small groups of engineers before being widely discussed. But learning how to distinguish the security bug reports from other bug reports is challenging since they may occur rarely. Data mining methods that can find such scarce targets require extensive optimization effort. Goal: The goal of this research is to aid practitioners as they struggle to optimize methods that try to distinguish between rare security bug reports and other bug reports. Method: Our proposed method, called Swift, is a dual optimizer that optimizes both learner and pre-processor options. Since this is a large space of options, Swift uses a technique called epsilon-dominance that learns how to avoid operations that do not significantly improve performance. Result: When compared to recent state-of-the-art results (from FARSEC which is published in TSE'18), we find that the Swift's dual optimization of both pre-processor and learner is more useful than optimizing each of them individually. For example, in a study of security bug reports from the Chromium dataset, the median recalls of FARSEC and Swift were 15.7% and 77.4%, respectively. For another example, in experiments with data from the Ambari project, the median recalls improved from 21.5% to 85.7% (FARSEC to SWIFT). Conclusion: Overall, our approach can quickly optimize models that achieve better recalls than the prior state-of-the-art. These increases in recall are associated with moderate increases in false positive rates (from 8% to 24%, median). For future work, these results suggest that dual optimization is both practical and useful.