A Multi-Dataset Evaluation of Models for Automated Vulnerability Repair

TL;DR

This paper evaluates pre-trained language models CodeBERT and CodeT5 for automated vulnerability patching across multiple datasets and languages, highlighting their strengths and limitations in generalization and scalability.

Contribution

It provides a comprehensive benchmark of these models for vulnerability repair, revealing their performance differences and challenges in generalizing to unseen vulnerabilities.

Findings

CodeT5 captures complex vulnerability patterns better.

CodeBERT performs better with fragmented or sparse context.

Models struggle to generalize to unseen vulnerabilities.

Abstract

Software vulnerabilities pose significant security threats, requiring effective mitigation. While Automated Program Repair (APR) has advanced in fixing general bugs, vulnerability patching, a security-critical aspect of APR remains underexplored. This study investigates pre-trained language models, CodeBERT and CodeT5, for automated vulnerability patching across six datasets and four languages. We evaluate their accuracy and generalization to unknown vulnerabilities. Results show that while both models face challenges with fragmented or sparse context, CodeBERT performs comparatively better in such scenarios, whereas CodeT5 excels in capturing complex vulnerability patterns. CodeT5 also demonstrates superior scalability. Furthermore, we test fine-tuned models on both in-distribution (trained) and out-of-distribution (unseen) datasets. While fine-tuning improves in-distribution performance, models struggle to generalize to unseen data, highlighting challenges in robust vulnerability detection. This study benchmarks model performance, identifies limitations in generalization, and provides actionable insights to advance automated vulnerability patching for real-world security applications.