PATCHEVAL: A New Benchmark for Evaluating LLMs on Patching Real-World Vulnerabilities

TL;DR

This paper introduces PATCHEVAL, a comprehensive multilingual benchmark with 1,000 real-world vulnerabilities across multiple programming languages, designed to evaluate and improve large language models' ability to automatically patch software security flaws.

Contribution

The paper presents PATCHEVAL, a novel, diverse, and reproducible benchmark for evaluating LLMs on real-world vulnerabilities in multiple programming languages, addressing limitations of previous benchmarks.

Findings

LLMs show promising capabilities in vulnerability patching.

Benchmark reveals gaps in current LLM performance.

Runtime verification improves patch validation reliability.

Abstract

Software vulnerabilities are increasing at an alarming rate. However, manual patching is both time-consuming and resource-intensive, while existing automated vulnerability repair (AVR) techniques remain limited in effectiveness. Recent advances in large language models (LLMs) have opened a new paradigm for AVR, demonstrating remarkable progress. To examine the capability of LLMs in AVR, several vulnerability benchmarks have been proposed recently. However, they still suffer from key limitations of outdated vulnerabilities, limited language coverage, unreliable patch validation, and insufficient reproducibility. To overcome these challenges, we introduce PATCHEVAL, a multilingual benchmark for Go, JavaScript, and Python, languages for which existing benchmarks remain unexplored. PATCHEVAL curates a dataset of 1,000 vulnerabilities drawn from CVEs reported between 2015 and 2025, covering 65 distinct CWEs. A subset of 230 CVEs is further equipped with runtime sandbox environments, enabling patch verification through both security tests and functionality tests. To provide a systematic comparison of LLM-based vulnerability repair, we evaluate a series of state-of-the-art LLMs and agents, presenting an in-depth analysis that empirically yields key insights to guide future research in AVR.