How the Training Procedure Impacts the Performance of Deep Learning-based Vulnerability Patching

TL;DR

This study systematically compares training procedures for deep learning models in vulnerability patching, revealing supervised pre-training's superiority and the effectiveness of prompt-tuning for self-supervised models.

Contribution

It provides the first comprehensive comparison of pre-training methods and explores prompt-tuning's impact on vulnerability patching performance.

Findings

Supervised pre-training on bug-fixing data significantly improves patching performance.

Prompt-tuning enhances self-supervised models without additional data collection.

No significant performance gain from prompt-tuning on supervised pre-trained models.

Abstract

Generative deep learning (DL) models have been successfully adopted for vulnerability patching. However, such models require the availability of a large dataset of patches to learn from. To overcome this issue, researchers have proposed to start from models pre-trained with general knowledge, either on the programming language or on similar tasks such as bug fixing. Despite the efforts in the area of automated vulnerability patching, there is a lack of systematic studies on how these different training procedures impact the performance of DL models for such a task. This paper provides a manyfold contribution to bridge this gap, by (i) comparing existing solutions of self-supervised and supervised pre-training for vulnerability patching; and (ii) for the first time, experimenting with different kinds of prompt-tuning for this task. The study required to train/test 23 DL models. We found that a supervised pre-training focused on bug-fixing, while expensive in terms of data collection, substantially improves DL-based vulnerability patching. When applying prompt-tuning on top of this supervised pre-trained model, there is no significant gain in performance. Instead, prompt-tuning is an effective and cheap solution to substantially boost the performance of self-supervised pre-trained models, i.e., those not relying on the bug-fixing pre-training.