Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Tasks

TL;DR

This paper benchmarks the security vulnerabilities of agent-generated code in vibe coding, revealing that most solutions are functionally correct but insecure, raising concerns about safety in real-world deployment.

Contribution

It introduces SU S VI B E S, a new benchmark for evaluating security vulnerabilities in vibe coding, and assesses the security of popular coding agents on real-world tasks.

Findings

All evaluated agents perform poorly in security.

61% of solutions are functionally correct, but only 10.5% are secure.

Security strategies like vulnerability hints do not effectively mitigate issues.

Abstract

Vibe coding is a new programming paradigm in which human engineers instruct large language model (LLM) agents to complete complex coding tasks with little supervision. Although vibe coding is increasingly adopted, are its outputs really safe to deploy in production? To answer this question, we propose SU S VI B E S, a benchmark consisting of 200 feature-request software engineering tasks from real-world open-source projects, which, when given to human programmers, led to vulnerable implementations. We evaluate multiple widely used coding agents with frontier models on this benchmark. Disturbingly, all agents perform poorly in terms of software security. Although 61% of the solutions from SWE-Agent with Claude 4 Sonnet are functionally correct, only 10.5% are secure. Further experiments demonstrate that preliminary security strategies, such as augmenting the feature request with vulnerability hints, cannot mitigate these security issues. Our findings raise serious concerns about the widespread adoption of vibe-coding, particularly in security-sensitive applications.