Shallow or Deep? An Empirical Study on Detecting Vulnerabilities using Deep Learning

TL;DR

This study empirically compares deep learning and shallow machine learning models for software vulnerability detection across multiple datasets and code representations, revealing that shallow models remain competitive and current DL models lack reliability.

Contribution

It provides a large-scale empirical evaluation of DL versus shallow models for vulnerability detection, highlighting the limited effectiveness of DL and the competitiveness of shallow classifiers.

Findings

Shallow classifiers perform competitively with DL models.

DL models are not yet reliably effective for vulnerability detection.

Current models still have significant room for improvement.

Abstract

Deep learning (DL) techniques are on the rise in the software engineering research community. More and more approaches have been developed on top of DL models, also due to the unprecedented amount of software-related data that can be used to train these models. One of the recent applications of DL in the software engineering domain concerns the automatic detection of software vulnerabilities. While several DL models have been developed to approach this problem, there is still limited empirical evidence concerning their actual effectiveness especially when compared with shallow machine learning techniques. In this paper, we partially fill this gap by presenting a large-scale empirical study using three vulnerability datasets and five different source code representations (i.e., the format in which the code is provided to the classifiers to assess whether it is vulnerable or not) to compare the effectiveness of two widely used DL-based models and of one shallow machine learning model in (i) classifying code functions as vulnerable or non-vulnerable (i.e., binary classification), and (ii) classifying code functions based on the specific type of vulnerability they contain (or "clean", if no vulnerability is there). As a baseline we include in our study the AutoML utility provided by the Google Cloud Platform. Our results show that the experimented models are still far from ensuring reliable vulnerability detection, and that a shallow learning classifier represents a competitive baseline for the newest DL-based models.