RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories

TL;DR

This paper introduces RealSec-bench, a benchmark based on real-world Java repositories to evaluate LLMs' ability to generate secure code, revealing current models' limitations in balancing security and functionality.

Contribution

We present a novel benchmark constructed from real-world repositories, combining systematic vulnerability detection and expert validation, to evaluate secure code generation in LLMs.

Findings

RAG improves functional correctness but not security.

Prompting with security guidelines often causes compilation failures.

Current LLMs struggle to generate secure code while maintaining functionality.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in code generation, but their proficiency in producing secure code remains a critical, under-explored area. Existing benchmarks often fall short by relying on synthetic vulnerabilities or evaluating functional correctness in isolation, failing to capture the complex interplay between functionality and security found in real-world software. To address this gap, we introduce RealSec-bench, a new benchmark for secure code generation meticulously constructed from real-world, high-risk Java repositories. Our methodology employs a multi-stage pipeline that combines systematic SAST scanning with CodeQL, LLM-based false positive elimination, and rigorous human expert validation. The resulting benchmark contains 105 instances grounded in real-word repository contexts, spanning 19 Common Weakness Enumeration (CWE) types and exhibiting a wide diversity of data flow complexities, including vulnerabilities with up to 34-hop inter-procedural dependencies. Using RealSec-bench, we conduct an extensive empirical study on 5 popular LLMs. We introduce a novel composite metric, SecurePass@K, to assess both functional correctness and security simultaneously. We find that while Retrieval-Augmented Generation (RAG) techniques can improve functional correctness, they provide negligible benefits to security. Furthermore, explicitly prompting models with general security guidelines often leads to compilation failures, harming functional correctness without reliably preventing vulnerabilities. Our work highlights the gap between functional and secure code generation in current LLMs.