On the Evaluation of Large Language Models in Multilingual Vulnerability Repair

TL;DR

This paper presents a large-scale empirical study evaluating the effectiveness of large language models, especially GPT-4o, in repairing software vulnerabilities across seven programming languages, highlighting their potential and limitations.

Contribution

It is the first comprehensive study comparing LLMs and existing approaches for multilingual vulnerability repair, demonstrating GPT-4o's competitive performance and generalization capabilities.

Findings

GPT-4o performs competitively with VulMaster.

LLMs are more effective in repairing unique and dangerous vulnerabilities.

Go language shows the highest repair effectiveness.

Abstract

Various Deep Learning-based approaches with pre-trained language models have been proposed for automatically repairing software vulnerabilities. However, these approaches are limited to a specific programming language (C/C++). Recent advances in large language models (LLMs) offer language-agnostic capabilities and strong semantic understanding, exhibiting potential to overcome multilingual vulnerability limitations. Although some work has begun to explore LLMs' repair performance, their effectiveness is unsatisfactory. To address these limitations, we conducted a large-scale empirical study to investigate the performance of automated vulnerability repair approaches and state-of-the-art LLMs across seven programming languages. Results show GPT-4o, instruction-tuned with few-shot prompting, performs competitively against the leading approach, VulMaster. Additionally, the LLM-based approach shows superior performance in repairing unique vulnerabilities and is more likely to repair the most dangerous vulnerabilities. Instruction-tuned GPT-4o demonstrates strong generalization on vulnerabilities in previously unseen language, outperforming existing approaches. Analysis shows Go consistently achieves the highest effectiveness across all model types, while C/C++ performs the worst. Based on findings, we discuss the promise of LLM on multilingual vulnerability repair and the reasons behind LLM's failed cases. This work takes the first look at repair approaches and LLMs across multiple languages, highlighting the promising future of adopting LLMs for multilingual vulnerability repair.