DUALGUAGE: Automated Joint Security-Functionality Benchmarking for Secure Code Generation

TL;DR

DUALGAUGE is an automated framework that jointly evaluates the security and correctness of code generated by large language models, addressing a critical gap in secure software development.

Contribution

It introduces DUALGAUGE, the first fully automated benchmark for simultaneous security and correctness evaluation of LLM-generated code, along with a curated dataset DUALGAUGE-BENCH.

Findings

Identified significant gaps in security and correctness in current LLMs

Provided a scalable, reproducible evaluation system for secure code generation

Published open-source tools and datasets to advance research in secure AI coding

Abstract

Large language models (LLMs) and autonomous coding agents are increasingly used to generate software across a wide range of domains. Yet a core requirement remains unmet: ensuring that generated code is secure without compromising its functional correctness. Existing benchmarks and evaluations for secure code generation fall short-many measure only vulnerability reduction, disregard correctness preservation, or evaluate security and functionality on separate datasets, violating the fundamental need for simultaneous joint evaluation. We present DUALGAUGE, the first fully automated benchmarking framework designed to rigorously evaluate the security and correctness of LLM-generated code in unison. Given the lack of datasets enabling joint evaluation of secure code generation, we also present DUALGAUGE-BENCH, a curated benchmark suite of diverse coding tasks, each paired with manually validated test suites for both security and functionality, designed for full coverage of specification requirements. At the core of DUALGAUGE is an agentic program executor, which runs a program against given tests in sandboxed environments, and an LLM-based evaluator, which assesses both correctness and vulnerability behavior against expected outcomes. We rigorously evaluated and ensured the quality of DUALGAUGE-BENCH and the accuracy of DUALGAUGE, and applied DUALGAUGE to benchmarking ten leading LLMs on DUALGAUGE-BENCH across thousands of test scenarios. Our results reveal critical gaps in correct and secure code generation by these LLMs, for which our open-source system and datasets help accelerate progress via reproducible, scalable, and rigorous evaluation.