EaTVul: ChatGPT-based Evasion Attack Against Software Vulnerability Detection

TL;DR

This paper introduces EaTVul, a novel adversarial attack method using ChatGPT to exploit vulnerabilities in deep learning-based software vulnerability detection, achieving high success rates and highlighting the need for robust defenses.

Contribution

The study presents EaTVul, a six-stage attack framework leveraging ChatGPT and genetic algorithms to generate effective adversarial examples against vulnerability detection models.

Findings

EaTVul achieves over 83% attack success rate with snippet size > 2.

With snippet size 4, EaTVul reaches 100% success rate.

The research underscores the importance of developing defenses against adversarial attacks.

Abstract

Recently, deep learning has demonstrated promising results in enhancing the accuracy of vulnerability detection and identifying vulnerabilities in software. However, these techniques are still vulnerable to attacks. Adversarial examples can exploit vulnerabilities within deep neural networks, posing a significant threat to system security. This study showcases the susceptibility of deep learning models to adversarial attacks, which can achieve 100% attack success rate (refer to Table 5). The proposed method, EaTVul, encompasses six stages: identification of important samples using support vector machines, identification of important features using the attention mechanism, generation of adversarial data based on these features using ChatGPT, preparation of an adversarial attack pool, selection of seed data using a fuzzy genetic algorithm, and the execution of an evasion attack. Extensive experiments demonstrate the effectiveness of EaTVul, achieving an attack success rate of more than 83% when the snippet size is greater than 2. Furthermore, in most cases with a snippet size of 4, EaTVul achieves a 100% attack success rate. The findings of this research emphasize the necessity of robust defenses against adversarial attacks in software vulnerability detection.