SeCodePLT: A Unified Platform for Evaluating the Security of Code GenAI

TL;DR

SeCodePLT is a comprehensive, scalable benchmark platform for evaluating the security risks and capabilities of code-generating LLMs, combining dynamic analysis with high-quality, diverse data.

Contribution

The paper introduces SeCodePLT, a novel benchmark framework that enhances coverage, data quality, and scale for security evaluation of code LLMs using dynamic metrics.

Findings

SeCodePLT covers 44 CWE-based risk categories across three programming languages.

It demonstrates broader coverage and higher data fidelity than existing benchmarks.

Evaluation reveals strengths and weaknesses of current code LLMs in security tasks.

Abstract

Existing benchmarks for evaluating the security risks and capabilities (e.g., vulnerability detection) of code-generating large language models (LLMs) face several key limitations: (1) limited coverage of risk and capabilities; (2) reliance on static evaluation metrics such as LLM judgments or rule-based detection, which lack the precision of dynamic analysis; and (3) a trade-off between data quality and benchmark scale. To address these challenges, we introduce a general and scalable benchmark construction framework that begins with manually validated, high-quality seed examples and expands them via targeted mutations. Our approach provides a comprehensive suite of artifacts so the benchmark can support comprehensive risk assessment and security capability evaluation using dynamic metrics. By combining expert insights with automated generation, we strike a balance between manual effort, data quality, and benchmark scale. Applying this framework to Python, C/C++, and Java, we build SeCodePLT, a dataset of more than 5.9k samples spanning 44 CWE-based risk categories and three security capabilities. Compared with state-of-the-art benchmarks, SeCodePLT offers broader coverage, higher data fidelity, and substantially greater scale. We use SeCodePLT to evaluate leading code LLMs and agents, revealing their strengths and weaknesses in both generating secure code and identifying or fixing vulnerabilities.