When Machine Learning Meets Vulnerability Discovery: Challenges and Lessons Learned

TL;DR

This paper discusses the challenges of applying machine learning to software vulnerability discovery, highlighting issues with dataset transparency, model choices, and evaluation, while sharing insights from two related research tools.

Contribution

It identifies key challenges in ML-based vulnerability detection and offers insights from previous tools to guide future research in the field.

Findings

Lack of detailed dataset statistics hampers evaluation.

Training on semantically similar functions raises concerns.

Model choice and granularity significantly impact effectiveness.

Abstract

In recent years, machine learning has demonstrated impressive results in various fields, including software vulnerability detection. Nonetheless, using machine learning to identify software vulnerabilities presents new challenges, especially regarding the scale of data involved, which was not a factor in traditional methods. Consequently, in spite of the rise of new machine-learning-based approaches in that space, important shortcomings persist regarding their evaluation. First, researchers often fail to provide concrete statistics about their training datasets, such as the number of samples for each type of vulnerability. Moreover, many methods rely on training with semantically similar functions rather than directly on vulnerable programs. This leads to uncertainty about the suitability of the datasets currently used for training. Secondly, the choice of a model and the level of granularity at which models are trained also affect the effectiveness of such vulnerability discovery approaches. In this paper, we explore the challenges of applying machine learning to vulnerability discovery. We also share insights from our two previous research papers, Bin2vec and BinHunter, which could enhance future research in this field.