Evaluating Large Language Models for Multilingual Vulnerability Detection at Dual Granularities

TL;DR

This study empirically evaluates large language models for multilingual vulnerability detection at different granularities, demonstrating GPT-4o's superior performance and highlighting the potential of LLMs in real-world security applications.

Contribution

It provides a comprehensive, fine-grained comparison of PLMs and LLMs across multiple programming languages and detection granularities, revealing LLMs' advantages in security tasks.

Findings

GPT-4o outperforms other models including CodeT5P.

LLMs excel in detecting multilingual and high-severity vulnerabilities.

Multilingual LLM-based vulnerability detection shows significant promise.

Abstract

Various deep learning-based approaches utilizing pre-trained language models (PLMs) have been proposed for automated vulnerability detection. With recent advancements in large language models (LLMs), several studies have begun exploring their application to vulnerability detection tasks. However, existing studies primarily focus on specific programming languages (e.g., C/C++) and function-level detection, leaving the strengths and weaknesses of PLMs and LLMs in multilingual and multi-granularity scenarios largely unexplored. To bridge this gap, we conduct a comprehensive fine-grained empirical study evaluating the effectiveness of state-of-the-art PLMs and LLMs for multilingual vulnerability detection. Using over 30,000 real-world vulnerability-fixing patches across seven programming languages, we systematically assess model performance at both the function-level and line-level. Our key findings indicate that GPT-4o, enhanced through instruction tuning and few-shot prompting, significantly outperforms all other evaluated models, including CodeT5P. Furthermore, the LLM-based approach demonstrates superior capability in detecting unique multilingual vulnerabilities, particularly excelling in identifying the most dangerous and high-severity vulnerabilities. These results underscore the promising potential of adopting LLMs for multilingual vulnerability detection at function-level and line-level, revealing their complementary strengths and substantial improvements over PLM approaches. This empirical evaluation of PLMs and LLMs for multilingual vulnerability detection highlights LLMs' value in addressing real-world software security challenges.