SeCodePLT: A Unified Platform for Evaluating the Security of Code GenAI
Yuzhou Nie, Zhun Wang, Yu Yang, Ruizhe Jiang, Yuheng Tang, Xander Davies, Yarin Gal, Bo Li, Wenbo Guo, Dawn Song

TL;DR
SeCodePLT is a comprehensive, scalable benchmark platform for evaluating the security risks and capabilities of code-generating LLMs, combining dynamic analysis with high-quality, diverse data.
Contribution
The paper introduces SeCodePLT, a novel benchmark framework that enhances coverage, data quality, and scale for security evaluation of code LLMs using dynamic metrics.
Findings
SeCodePLT covers 44 CWE-based risk categories across three programming languages.
It demonstrates broader coverage and higher data fidelity than existing benchmarks.
Evaluation reveals strengths and weaknesses of current code LLMs in security tasks.
Abstract
Existing benchmarks for evaluating the security risks and capabilities (e.g., vulnerability detection) of code-generating large language models (LLMs) face several key limitations: (1) limited coverage of risk and capabilities; (2) reliance on static evaluation metrics such as LLM judgments or rule-based detection, which lack the precision of dynamic analysis; and (3) a trade-off between data quality and benchmark scale. To address these challenges, we introduce a general and scalable benchmark construction framework that begins with manually validated, high-quality seed examples and expands them via targeted mutations. Our approach provides a comprehensive suite of artifacts so the benchmark can support comprehensive risk assessment and security capability evaluation using dynamic metrics. By combining expert insights with automated generation, we strike a balance between manual…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsSoftware Reliability and Analysis Research · Software Engineering Research · Scientific Computing and Data Management
MethodsFocus · Sparse Evolutionary Training
