A.S.E: A Repository-Level Benchmark for Evaluating Security in AI-Generated Code

TL;DR

This paper introduces A.S.E, a comprehensive benchmark for evaluating the security of AI-generated code at the repository level, addressing the gap in real-world applicability of existing benchmarks.

Contribution

The paper presents A.S.E, a novel repository-level benchmark that better reflects real-world AI programming scenarios for security evaluation of generated code.

Findings

Current LLMs struggle with secure coding in repository scenarios.

Larger reasoning budgets do not improve code security.

Repository complexity challenges LLM performance.

Abstract

The increasing adoption of large language models (LLMs) in software engineering necessitates rigorous security evaluation of their generated code. However, existing benchmarks often lack relevance to real-world AI-assisted programming scenarios, making them inadequate for assessing the practical security risks associated with AI-generated code in production environments. To address this gap, we introduce A.S.E (AI Code Generation Security Evaluation), a repository-level evaluation benchmark designed to closely mirror real-world AI programming tasks, offering a comprehensive and reliable framework for assessing the security of AI-generated code. Our evaluation of leading LLMs on A.S.E reveals several key findings. In particular, current LLMs still struggle with secure coding. The complexity in repository-level scenarios presents challenges for LLMs that typically perform well on snippet-level tasks. Moreover, a larger reasoning budget does not necessarily lead to better code generation. These observations offer valuable insights into the current state of AI code generation and help developers identify the most suitable models for practical tasks. They also lay the groundwork for refining LLMs to generate secure and efficient code in real-world applications.