Can Highlighting Help GitHub Maintainers Track Security Fixes?

TL;DR

This paper explores whether highlighting techniques can assist GitHub maintainers in tracking security fixes by evaluating explainable AI methods like LIME and TfIdf-Highlight for improving patch retrieval and decision-making.

Contribution

The paper introduces TfIdf-Highlight, a novel explanation method for security patch retrieval, and compares its effectiveness with LIME in aiding maintainers.

Findings

TfIdf-Highlight outperforms LIME in faithfulness scores by 15%.

Highlighting increases helpfulness scores but does not improve labeling accuracy.

Both LIME and TfIdf-Highlight show similar accuracy in human experiments.

Abstract

In recent years, the rapid growth of security vulnerabilities poses great challenges to tracing and managing them. For example, it was reported that the NVD database experienced significant delays due to the shortage of maintainers. Such delay creates challenges for third-party security personnel (e.g., administrators) to trace the information related to the CVE. To help security personnel trace a vulnerability patch, we build a retrieval system that automatically retrieves the patch in the repository. Inspired by existing work on explainable machine learning, we ask the following research question: can explanations help security maintainers make decisions in patch tracing? First, we investigate using LIME (a widely used explainable machine learning method) to highlight the rationale tokens in the commit message and code. In addition, we propose an explanation method called TfIdf-Highlight, which leverages the Tf-Idf statistics to select the most informative words in the repository and the dataset. We evaluate the effectiveness of highlighting using two experiments. First, we compare LIME and TfIdf-Highlight using a faithfulness score (i.e., sufficiency and comprehensiveness) defined for ranking. We find that TfIdf-Highlight significantly outperforms LIME's sufficiency scores by 15\% and slightly outperforms the comprehensiveness scores. Second, we conduct a blind human labeling experiment by asking the annotators to guess the patch under 3 settings (TfIdf-Highlight, LIME, and no highlight). We find that the helpfulness score for TfIdf-Highlight is higher than LIME while the labeling accuracies of LIME and TfIdf-Highlight are similar. Nevertheless, highlighting does not improve the accuracy over non-highlighting.