Evaluating Pre-Trained Models for Multi-Language Vulnerability Patching

TL;DR

This paper evaluates pre-trained models CodeBERT and CodeT5 for automated vulnerability patching across multiple languages, highlighting their strengths, limitations, and potential for security applications.

Contribution

It provides a comprehensive benchmark of these models' performance on vulnerability patching, revealing insights into their accuracy, efficiency, and challenges with patch length.

Findings

CodeT5 achieves higher accuracy on complex vulnerability datasets.

CodeBERT performs well with fragmented or limited context data.

Both models struggle as patch length increases.

Abstract

Software vulnerabilities pose critical security risks, demanding prompt and effective mitigation strategies. While advancements in Automated Program Repair (APR) have primarily targeted general software bugs, the domain of vulnerability patching, which is a security-critical subset of APR, remains underexplored. This paper investigates the potential of pre-trained language models, CodeBERT and CodeT5, for automated vulnerability patching across diverse datasets and five programming languages. We evaluate these models on their accuracy, computational efficiency, and how the length of vulnerable code patches impacts performance. Our findings reveal promising accuracy levels, particularly for CodeT5 on datasets with complex vulnerability patterns, while CodeBERT demonstrates strengths in handling fragmented or context-limited datasets. CodeT5 further showcases superior efficiency, making it well-suited for large-scale applications. However, both models face challenges in maintaining performance as patch length increases, highlighting the complexity of addressing extended in program repair specifically aimed at fixing vulnerabilities. This study benchmarks model performance, highlights key limitations, and offers insights to improve automated vulnerability patching for practical security applications.