AVIATOR: Towards AI-Agentic Vulnerability Injection Workflow for High-Fidelity, Large-Scale Code Security Dataset

TL;DR

AVIATOR is a novel AI-driven vulnerability injection framework that enhances dataset quality for software security by achieving high fidelity and coverage, significantly improving deep learning-based vulnerability detection.

Contribution

It introduces the first AI-agentic vulnerability injection workflow that uses specialized AI agents and lightweight fine-tuning to produce realistic, category-specific vulnerabilities without handcrafted patterns.

Findings

Achieves 91-95% injection fidelity, surpassing existing methods.

Improves deep learning vulnerability detection F1 scores by +22% to +25%.

Scales efficiently with low rejection rates and reduced costs.

Abstract

The increasing complexity of software systems and the sophistication of cyber-attacks have underscored the need for reliable automated software vulnerability detection. Data-driven approaches using deep learning models show promise but critically depend on the availability of large, accurately labeled datasets. Yet existing datasets either suffer from noisy labels, limited vulnerability coverage, or fail to reflect vulnerabilities as they occur in real-world software. This also limits large-scale benchmarking of such solutions. Automated vulnerability injection provides a way to address these limitations, but existing techniques remain limited in coverage, contextual fidelity, or injection success. In this paper, we present AVIATOR, the first AI-agentic vulnerability injection framework. AVIATOR decomposes vulnerability injection into a coordinated workflow of specialized AI agents, tool-based analysis, and iterative self-correction, explicitly mirroring expert reasoning. It integrates RAG and lightweight LoRA-based fine-tuning to produce realistic, category-specific vulnerabilities without relying on handcrafted patterns. Across three benchmarks, AVIATOR achieves high injection fidelity (91-95%) surpassing existing injection techniques in both accuracy and vulnerability coverage. When used for data augmentation to train deep learning-based vulnerability detection (DLVD) models, AVIATOR provides the strongest downstream gains in vulnerability detection. Across models and base datasets, AVIATOR improves average F1 scores by +22% over no augmentation, +25% over VGX, holding the prior best injection success rate, and +3% over VulScribeR, the prior state-of-the-art LLM-based injection model, with +7% higher recall and no precision loss. Its augmented data exhibits the lowest distributional distortion and scales efficiently with <2% syntax rejection at 4.3x lower cost than VulScribeR.

Tables

Table 1. TABLE I: Statistics of the Studied Datasets

Dataset	Vulnerable Samples	Clean Samples	Total	Ratio	CWE Num	Entropy
SARD100 [22]	102	102	204	1:1	23	3.98
FormAI [48]	192,712	117,774	310,486	1:0.6	19	3.61
PrimeVul Train [12]	4,802	169,345	174,147	1:35.3	122	4.76
PrimeVul Validation	593	23,347	23,940	1:39.4	73	5.05
PrimeVul Test	549	24,232	24,781	1:44.1	71	5.09
Devign [56]	10,768	12,024	22,792	1:1.1	NA	4.84
BigVul Train [17]	8,783	142,125	150,908	1:16.2	91	4.94
BigVul Validation	1,038	17,826	18,864	1:17.2	84	4.95
BigVul Test	1,079	17,785	18,864	1:16.5	85	4.72

Table 2. TABLE II: AVIATOR injection success across datasets: Average Injection Success Rate on 5 runs for SARD100 and FormAI, and on analyzable cases for PrimeVul due to limited context.

Dataset	Eval. Type	Metric	AVIATOR
SARD100	Automated	$AIS{R}_5$	95%
FormAI	Automated	$AIS{R}_5$	91%
PrimeVul	Manual	% Analyzable Weaknesses	94%

Table 3. TABLE III: Label accuracy and injection success rates (marked with *) on major vulnerability datasets satisfying the BCD properties (Section  II-A ). Results for AVIATOR are obtained via automated evaluation on SARD100 and FormAI, and manual validation on PrimeVul. The reported score for AVIATOR corresponds to the mean injection success rate across all three evaluation datasets.

Model / Dataset (*)	Accuracy	Source
Graph2Edit * [53]	13%	[37]
BigVul [17]	25%	[12]
AVIATOR $W_1$ *	31%	FormAI validation
CrossVul [35]	48%	[12]
Getafix * [53]	50%	[37]
CVEFixes [3]	52%	[12]
DiverseVul [7]	60%	[12]
VulGen * [37]	69%	[37]
VinJ * [38]	69%	[38]
VulScribeR * [10]	82%	[10]
VGX * [36]	90%	[36]
AVIATOR $W_{13}$ & SFT *	93%	Cross-Dataset Eval.

Table 4. TABLE IV: AVIATOR W 13 W_{13} injection success rate ( A ​ I ​ S ​ R 5 AISR_{5} ) across 5 vulnerability injection generation with and without fine-tuning, on FormAI and SARD100.

Dataset	Model Variant	$AIS{R}_5$	Std
FormAI	Qwen2.5 (no FT)	85%	4
	Qwen2.5 + GRPO	84%	1
	Qwen2.5 + SFT	91%	1
SARD100	Qwen2.5 (no FT)	94%	3
	Qwen2.5 + GRPO	94%	0
	Qwen2.5 + SFT	95%	2

Table 5. TABLE V: AVIATOR W 13 W_{13} pass@k for 10 vulnerability injection generations with and without fine-tuning, on FormAI and SARD100.

Dataset	Model	@1	@2	@3	@4	@5	@6	@7	@8	@9	@10
FormAI	$W_{13}$	84.32%	91.05%	93.87%	95.24%	96.03%	96.56%	96.94%	97.18%	97.30%	97.30%
	$W_{13}$ + SFT	89.91%	93.04%	94.81%	95.62%	96.15%	96.55%	96.87%	97.14%	97.38%	97.58%
SARD100	$W_{13}$	93.58%	95.95%	96.66%	96.91%	97.01%	97.05%	97.06%	97.06%	97.06%	97.06%
	$W_{13}$ + SFT	95.15%	98.65%	99.51%	99.79%	99.93%	99.99%	100.00%	100.00%	100.00%	100.00%

Table 6. TABLE VI: Downstream vulnerability detection performance (Precision, Recall, F1). Each model is trained on the seed dataset augmented with 5k injected vulnerable samples using each strategy. Best F1 scores per model and setting are highlighted in bold . Results for non-AVIATOR strategies are taken from the VulScribeR paper [ 10 ] ; we report new results for AVIATOR and CodeT5.

Strategy	Devign Model			Reveal Model			LineVul Model			CodeT5 Model
	P	R	F1	P	R	F1	P	R	F1	P	R	F1
Seed Dataset: Devign; Test Dataset: BigVul Test
No Aug	6.27	43.24	10.95	7.28	77.27	13.31	7.90	29.19	12.43	6.08	65.71	11.13
ROS	7.54	25.60	11.65	7.96	33.39	12.86	10.91	13.16	11.93	7.52	51.81	13.13
VulGen	6.49	62.32	11.75	6.36	70.59	11.66	8.51	22.15	12.29	7.34	31.05	11.87
VGX	6.06	59.30	10.99	6.89	73.29	12.60	5.35	98.82	10.15	7.20	80.17	13.21
VulScribeR	8.79	29.41	13.53	11.64	38.00	17.82	11.43	18.81	14.22	7.09	76.09	12.97
AVIATOR	9.59	24.80	13.83	9.19	56.94	15.82	11.27	29.99	16.38	8.08	65.89	14.39
Seed Dataset: PrimeVul Train; Test Dataset: PrimeVul Test
No Aug	25.93	5.12	8.55	7.35	44.63	12.62	18.31	31.15	23.06	19.34	25.68	22.07
ROS	7.45	52.44	13.04	7.58	43.66	13.07	18.48	30.05	22.89	18.03	28.42	22.07
VulGen	4.94	59.76	9.11	5.29	59.27	9.72	19.39	28.23	22.99	19.82	24.04	21.73
VGX	5.61	53.66	10.15	5.44	50.73	9.82	19.77	27.69	23.06	16.61	32.79	22.05
VulScribeR	7.57	55.37	13.31	7.81	49.76	13.51	17.29	31.88	22.42	20.26	25.14	22.44
AVIATOR	7.73	51.82	13.45	8.65	36.32	13.97	19.74	28.05	23.18	16.96	31.33	22.01

Table 7. TABLE VII: Overall Average Relative Improvement of AVIATOR Compared to Other Strategies.

Compared Strategy	Precision	Recall	F1-Score
No Aug	+12.2%	+104.9%	+21.8%
ROS	+9.0%	+26.0%	+12.5%
VulGen	+30.3%	+5.7%	+25.2%
VGX	+39.2%	-25.4%	+24.6%
VulScribeR	+1.4%	+7.4%	+2.9%

Equations

$$\mathsf{inject}:T^n\times V⟶T^m,(c^b,v_j)\mapsto c^{\mathrm{v}}$$

$${\tau }_i:T⟶T^{\ast },{\tau }_i(c_{(i)}^b)=⟨t_1^{(i)},\dots ,t_{m^{(i)}}^{(i)}⟩,$$

$$\text{For }{m}^{(i)}⩾1;\{\begin{array}{ll}{\tau }_i(c_{(i)}^b)=⟨t_1^{(i)},\dots ,t_{m^{(i)}}^{(i)},c_{(i)}^b⟩,& \text{or}\\ {\tau }_i(c_{(i)}^b)=⟨c_{(i)}^b,t_1^{(i)},\dots ,t_{m^{(i)}}^{(i)}⟩.& \end{array}$$

$${\tau }_i(c_{(i)}^b)=⟨⟩,m^{(i)}=0.$$

$${\tau }_i(c_{(i)}^b)=⟨t_1^{(i)},\dots ,t_{m^{(i)}}^{(i)}⟩,m^{(i)}⩾1.$$

$$c^v=\underset{i=1}{\overset{n}{⨁}}{\tau }_i(c_{(i)}^b)=⟨c_1^{\prime },c_2^{\prime },\dots ,c_m^{\prime }⟩,$$

$$\mathcal{Q}={\left\{\left(c^{b_{(j)}},{\hat{c}}^{v_{(j)}}\right)\right\}}_{j=1}^N$$

$${\mathcal{L}}_{\text{SFT}}(\varphi )=-\frac{1}{B}\sum _{i=1}^B\sum _{t=0}^{T_i-1}\log P\left(y_{i,t+1}\right|{\mathbf{x}}_i,y_{i,1},\dots ,y_{i,t};\varphi ).$$

$$\begin{gathered} \text{CodeBLEU}=\alpha \cdot \text{BLEU}(n\text{-grams})+\beta \cdot {\text{BLEU}}_w(n\text{-grams}) \\ +\gamma \cdot {\text{Match}}_{\text{AST}}+\delta \cdot {\text{Match}}_{\text{Data-flow},} \end{gathered}$$

$$A_i=\frac{r(o_i)-\mu }{\sigma },\mu =\frac{1}{G}\sum _{j=1}^{G}r(o_j),\sigma =\text{std}\left(\{r(o_j){\}}_{j=1}^G\right).$$

$$\begin{gathered} {\mathcal{J}}_{\text{GRPO}}(\theta )=\mathbb{E}[\frac{1}{G}\sum _{i=1}^G(\min (r_i{A}_i,\text{clip}(r_i,1-ϵ,1+ϵ)A_i) \\ -\beta {\mathcal{D}}_{\text{KL}}({\pi }_{\theta }\parallel {\pi }_{\text{ref}}))] \end{gathered}$$

$${\mathcal{D}}_{\text{KL}}({\pi }_{\theta }\Vert {\pi }_{\text{ref}})=\sum _{o_i}{\pi }_{\text{ref}}(o_i|q)\log \frac{{\pi }_{\text{ref}}(o_i|q)}{{\pi }_{\theta }(o_i|q)}-1.$$

$$AIS{R}_n=\frac{1}{n}\sum _{i=1}^n{s}_i,\sigma =\sqrt{\frac{1}{n}\sum _{i=1}^n(s_i-\text{AISR}{)}^2},$$

$$\text{Pass@}k=1-\frac{\left(\genfrac{}{}{0ex}{}{n-c}{k}\right)}{\left(\genfrac{}{}{0ex}{}{n}{k}\right)},\text{if }c<k;\text{Pass@}k=1\text{ otherwise}$$

Full text

I Introduction

The rapid growth in software complexity, coupled with the sophistication of cyber-attacks, poses a significant threat to the global security and stability of digital infrastructures. In 2024 alone, the total number of publicly reported vulnerabilities rose by 25% [8]. Software vulnerabilities refer to weaknesses in system security requirements, design, implementation, or operation, that could be accidentally triggered or intentionally exploited, resulting in a violation of the system’s security policy [11]. These trends highlight the urgent need for scalable and effective automated vulnerability detection and repair systems. Traditional methods, such as static program analysis, have played an important role. However, their reliance on manually crafted heuristics and the combinatorial complexity of their analysis make them fundamentally limited, with persistent challenges in scalability, adaptability, and false alarm rates [26, 14].

In response, AI-based approaches for code vulnerability detection [4, 55, 31, 30, 42, 7, 12] and repair [1, 18, 25] have recently attracted considerable attention in the research community. These data-driven techniques show strong potential in identifying and addressing a wide range of vulnerabilities. However, their effectiveness heavily relies on the quality and scale of the training data [12, 36]. This dependency has become even more critical with the emergence of Large Language Models (LLMs) with billions of parameters to train. For vulnerability detection in particular, the practical objective is not only to curate accurate labels, but to generate data that measurably improves downstream deep learning-based vulnerability detection (DLVD) under realistic test distributions. Yet high-quality vulnerable samples remain scarce [12, 55, 4], and existing datasets often trade off realism, label fidelity, and coverage, limiting downstream generalization.

Existing vulnerability datasets can be grouped into the following categories based on how the code samples were collected and annotated:

• •

Synthetic datasets: Suites such as Juliet [39] provide clean, controllable examples, but they rarely reflect the structural and semantic complexity of production code. Models trained on such data often learn superficial cues rather than vulnerability causes [4].

• •

Large realistic datasets: Datasets such as ReVeal [4], BigVul [17], CVEfixes [3], CrossVul [35], and DiverseVul [7] rely on heuristic labeling from vulnerability-fixing commits and security databases (e.g., NVD [33]). While they capture real code, heuristic labeling introduces substantial noise, with reported label accuracies as low as 25% for some datasets [12, 9]. Static-analyzer-labeled datasets (e.g., D2A [54], Draper [46]) suffer similarly from false positives due to tool limitations.

• •

Reliable-label datasets: Manual annotation (e.g., SVEN [21]) improves label fidelity but remains expensive and small-scale. Conversely, strict heuristics (e.g., PrimeVul [12]) improve reliability to human-level labeling accuracy, but can constrain scalability and coverage. Neither route offers a robust path to producing large, high-quality vulnerable data at scale.

• •

Vulnerability injection-based datasets: Injection aims to create vulnerable samples by introducing realistic weaknesses into existing code bases, which scales better than manual curation and typically yields more natural code than purely synthetic generators [41, 13, 44, 27, 10]. Recent systems such as VULGEN [37], VinJ [38], and VGX [36] combine pattern mining with deep learning-based localization to select injection sites and apply vulnerability-inducing edits. While these approaches can generate large volumes of samples, they remain constrained in both coverage and reliability. In practice, their edits often reduce to single-statement transformations and a limited set of Common Weakness Enumeration (CWE) types, which restricts semantic and contextual realism and can lead to brittle downstream benefits. VGX [36] reports higher injection success (about 90%) but supports only 23 CWE types and relies on a partly manual pattern-mining pipeline. In parallel, VulScribeR [10] leverages LLMs with RAG and explicitly optimizes for downstream DLVD performance rather than maximizing injection fidelity; it reports an estimated injection success of 80%, and its generation quality can still depend on model variability and post-filtering. Overall, existing injection methods either achieve high success under narrow, template-like edits (e.g., VGX) or improve downstream metrics without directly targeting high-fidelity, broad-coverage injection (e.g., VulScribeR).

To address these challenges, we introduce a novel AI agentic workflow designed to automatically inject realistic, diverse, category-specific vulnerabilities for large-scale generation of high-quality vulnerability datasets. This framework, called AVIATOR (AI Agentic Vulnerability Injection And Transformation with Optimized Reasoning) coordinates multiple AI agents, simulating expert reasoning, alongside functional agents and traditional code analysis tools. It was designed by studying the full process of manual injection and splitting it into elementary subtasks, relying on tools and self-correction loops to provide optimal guidance to LLMs. Its design was validated by three cybersecurity experts (including two co-authors). To further boost performance, it leverages Retrieval-Augmented Generation (RAG) [29] for contextual grounding and employs Low-Rank Adaptation (LoRA) [23] for efficient model fine-tuning. Crucially, AVIATOR requires no pattern mining, freeing it from the rigid constraints of mined edits and allowing scalable generation of vulnerabilities that span more CWE categories, with greater semantic and contextual realism.

To evaluate AVIATOR’s ability to introduce vulnerabilities, we conducted an experimental study on code samples from three widely used benchmarks [34, 48, 12]. We used the formal verification tool ESBMC [20] to validate supported vulnerabilities in smaller, self-contained functions, and we relied on manual evaluation by experienced security researchers for larger functions with missing project-level context, where ESBMC is not applicable. We compare AVIATOR with existing automated large-scale realistic software vulnerability dataset creation techniques. The results show that AVIATOR achieves highest injection fidelity, with success rates between 91% and 95% across the evaluated benchmarks.

Beyond injection fidelity, AVIATOR also delivers the strongest downstream gains under the same DLVD augmentation protocol as VulScribeR [10]. With 5k injected samples, AVIATOR improves average F1 score across models and datasets by +21.8% over no augmentation and by +24.6% over VGX, and it outperforms the prior injection-based downstream state of the art VulScribeR by +2.9% in average F1 score while increasing recall by +7.4%. Notably, these recall gains do not come at the expense of precision, which is consistent with cleaner injected vulnerabilities and a more learnable training signal. We further attribute this robustness to distributional fidelity: among all augmentation strategies, AVIATOR induces the smallest shift from the original real-world dataset (lowest KL divergence on CWE count and entropy distributions), indicating that it enriches training data without skewing it toward narrow templates. Finally, unlike VulScribeR, which explicitly optimizes downstream detection performance rather than injection correctness, AVIATOR targets correct vulnerability instantiation and validation, making it suitable not only for augmentation but also for constructing benchmark datasets with reliable labels.

AVIATOR is a generic framework capable of supporting injection across a large CWE spectrum. Our current instantiation covers 133 CWE types, far exceeding VGX (23 CWEs) and demonstrating the diversity potential of datasets generated with AVIATOR. Unlike prior injection-based approaches, AVIATOR does not rely on handcrafted pattern mining. Instead, its agentic design leverages LLM-based reasoning, making the workflow broadly applicable and easier to adapt across programming languages. In addition, AVIATOR achieves this quality efficiently: it maintains a $\mathbf{<2\%}$ syntax rejection rate and reduces generation cost by 4.3$\times$ compared to VulScribeR, enabling practical large-scale dataset construction.

The key contributions of this work are:

• •

AVIATOR: Agentic Vulnerability Injection Framework. To the best of our knowledge, this is the first work to explore vulnerability injection using an AI-agentic workflow. We carefully designed a multi-agent pipeline, leveraging tool-augmented subtasks with self-correction and validation loops, that replicates the reasoning processes of cybersecurity experts.

• •

High-Fidelity, Broad-Coverage Vulnerability Injection. Our evaluation demonstrates that AVIATOR achieves between 91% and 95% success in generating plausible vulnerabilities across three diverse benchmarks. AVIATOR is designed as a generic framework capable of supporting injection across a wide spectrum of weakness categories; our current instantiation supports 133 CWE types, far exceeding state-of-the-art systems like VGX. This demonstrates a strong potential for constructing diverse, high-coverage datasets.

• •

Downstream Effectiveness. Following the VulScribeR augmentation protocol, adding 5,000 AVIATOR-generated samples yields the strongest average downstream performance among all evaluated strategies. AVIATOR improves the average F1-score by +21.8% over no augmentation and +24.6% over VGX. Notably, it surpasses the LLM-based state-of-the-art VulScribeR by +2.9% in average F1 score and +7.4% in recall, achieving these gains without degrading precision.

• •

Realistic Augmentation and Practical Scalability. We show that AVIATOR induces the smallest distributional distortion relative to the original training set (lowest KL divergence) among augmentation strategies, indicating more realistic augmentation. AVIATOR also scales efficiently, maintaining a $\mathbf{<2\%}$ syntax rejection rate and achieving a 4.3$\times$ lower generation cost than the LLM-based technique VulScribeR. Through our experiments, we have established the defining characteristics of high-quality software vulnerability datasets, essential for both training AI models and benchmarking purposes.

The remainder of this paper is organized as follows: Section 2 reviews existing methods for vulnerability detection and data generation. Section 3 details the architecture and agentic workflow of AVIATOR. Section 4 outlines our experimental and evaluation setup. Section 5 reports empirical results and analysis. Section 6 discusses limitations and future work, and Section 7 concludes.

II Background and Motivation

In this section, we introduce the fundamental concepts and challenges associated with software vulnerability injection to motivate our proposed approach. We first describe the properties necessary for constructing effective datasets to train and benchmark Deep Learning-based vulnerability detection systems. Then, we formalize and illustrate the process of vulnerability injection in source code. Additionally, we clarify the scope and objectives that underpin our proposed solution.

II-A Properties of High-Quality Vulnerability Datasets

Effective training and rigorous benchmarking of DLVD methods critically depend on the quality of the underlying datasets. To clearly define the desired dataset characteristics, we introduce the ABCD criteria, similar to [11] and [13]:

• •

Accurate: Labels must reflect ground truth with clear information on the vulnerability type (CWE identifiers) and, when possible, precise vulnerability locations, to support more reliable training and faithful evaluation.

• •

Big: The dataset should be large enough to support training of deep learning models and enable statistically meaningful benchmarking of detection and repair systems.

• •

Credible: Code samples should reflect the structure and complexity of real-world software to ensure model generalization in practical settings.

• •

Diverse: The dataset should cover varied categories of vulnerability and coding styles to enable generalizable learning and comprehensive benchmarking across scenarios.

In practice, additional constraints improve training utility:

• •

Each sample should ideally contain a single vulnerability type to reduce ambiguity during training and evaluation.

• •

Paired benign and vulnerable samples, are valuable for repair tasks and contrastive learning.

II-B Motivation

Creating datasets that rigorously fulfill the ABCD criteria and structural requirements remains an open challenge due to significant limitations inherent to existing approaches [55]. Synthetic datasets, while accurate and easily scalable, lack credibility and offer limited diversity, as they rely on simplistic, predefined vulnerability patterns [4]. Heuristic-based labeling, in contrast, provides credible and diverse samples by mining real-world open-source repositories, but it exposes a persistent trade-off between accuracy and scalability: stricter heuristics yield smaller but cleaner datasets [12], while relaxed heuristics introduce noise [7]. Static analysis-based labeling can generate large datasets, yet accuracy remains a concern due to high false-positive rates. Manual annotation, though highly accurate, is not scalable for large datasets [28].

Vulnerability injection presents a promising alternative. By design, this data augmentation technique enables scalability through automation and preserves credibility when applied to real-world codebases. In principle, injection can also provide paired benign/vulnerable samples with explicit CWE labels and locations. However, the effectiveness of this approach depends heavily on the ability of the injection technique to introduce diverse and accurate vulnerabilities.

Existing injection systems have clear limitations. Pattern mining approaches such as VulGen and VinJ combine mined edit templates with deep learning-based localization to select injection sites in real code, which improves realism and scalability, but they still operate under constrained edit patterns and limited coverage, and produce noisy labels [37, 38]. VGX reports higher injection success, yet it remains tied to a partly manual pattern mining pipeline and focuses on single-statement edits with limited CWE support (23 CWE types), which narrows the space of vulnerabilities it can realistically generate [36]. In parallel, VulScribeR leverages LLMs and RAG to optimize downstream DLVD performance, but it does not explicitly target injection correctness and therefore provides noisier training signal and weaker guarantees for benchmark construction where label fidelity matters [10]. Overall, prior work tends to optimize either injection success under narrow templates (e.g., VGX) or downstream metrics without prioritizing injection fidelity (e.g., VulScribeR), leaving open the problem of achieving high fidelity injection, broad coverage, and strong downstream utility at scale.

II-C Vulnerability Injection

We consider a code snippet $c$ as an ordered sequence of lexical tokens $c=\langle c_{1},c_{2},\dots,c_{n}\rangle,\quad c_{i}\in T$, where $T$ is the vocabulary of the programming language (e.g., operators, delimiters, identifiers, literals, keywords, punctuation), and the token sequence conforms to the language grammar.

We define a vulnerability injection operator:

[TABLE]

which transforms a benign snippet $c^{b}$ into a vulnerable snippet $c^{v}$ exhibiting the vulnerability $v_{l}\in V$, where $V$ is the set of existing vulnerability types.

Injection is performed through a finite set of token-level edits. Let $I=\{\,i_{1},\dots,i_{k}\}\subseteq\{1,\dots,n\}$ be the set of positions to modify, and let each edit be defined by a function

[TABLE]

where $m^{(i)}\geq 0$, and $T^{*}$ is the set of finite token sequences. This formulation captures:

• •

Insertion: New tokens are inserted before or after $c^{b}_{(i)}$:

[TABLE]

• •

Deletion: $c^{b}_{(i)}$ is removed entirely:

[TABLE]

• •

Replacement: $c^{b}_{(i)}$ is replaced by a new non-empty token sequence:

[TABLE]

For positions $i\notin I$, we define $\tau_{i}(c^{b}_{(i)})=\langle c^{b}_{(i)}\rangle$, leaving the token unchanged.

The final injected sequence is obtained by flattening all transformed segments:

[TABLE]

where $\bigoplus$ denotes sequence concatenation.

A high-quality injection must satisfy the following properties:

• •

Realism: Edits $\{\tau_{i}\}$ should reflect patterns observed in real-world vulnerabilities of class $v_{j}$ (e.g. adapt to existing dataflow, meaningful output, no dead storage …).

• •

Functional Integrity: $c^{v}$ must remain syntactically and semantically valid, preserving the original functionality of $(c^{b})$, aside from the injected flaw.

• •

Minimality: The number of edits $|I|$ should be kept minimal.

III Proposed Framework

Existing vulnerability injection techniques trade off fidelity, coverage, and downstream utility. Pattern-mining methods such as VulGen, VinJ, and VGX achieve scalable injection but are constrained to narrow edit templates and limited CWE coverage, while recent LLM-based approaches such as VulScribeR prioritize downstream DLVD performance without explicitly enforcing injection correctness. To address these limitations, we propose AVIATOR (Figure 1), which formulates vulnerability injection as an AI-agentic workflow that explicitly mirrors the structured reasoning process of a human security analyst.

This section details the design principles underlying AVIATOR, the agentic workflow abstraction on which it is built, and the two core modules of the framework: vulnerability injection and validation.

III-A AI Agentic Workflows

AVIATOR builds on the concept of an agentic workflow, where a complex task is decomposed into smaller, well-defined subtasks executed by specialized agents. Each agent is instantiated as either a large language model (LLM) or a programmatic tool such as a static analyzer. Agents are connected through a directed execution graph with typed inputs and outputs, and conditional transitions that allow for retries or fallbacks when earlier steps fail. In contrast to monolithic program transformations that rely on a single model to perform injection end-to-end, this design constrains the scope of each agent and enables structured feedback between stages.

Formally, an agentic workflow $W_{k}$ with $k$ agents is a directed graph where nodes correspond to agents and edges represent conditional transitions. Each agent $A_{i}$ is defined as a transformation $A_{i}:X_{i}\times C_{i}\mapsto Y_{i}$, where $X_{i}$ is the structured input, $C_{i}$ is an optional set of contextual artifacts (e.g., CWE descriptions, retrieved examples, static analysis reports), and $Y_{i}$ is the output. Transitions are governed by conditions dynamically evaluated from prior outputs, allowing the workflow to re-invoke or bypass agents as needed.

The agentic workflow design provides advantages over monolithic reasoning or linear pipelines. First, it enables modular decomposition: the overall task is split into smaller, logically coherent subtasks aligned with the competencies of individual agents. This reduces the reasoning burden on any single component and improves interpretability. Second, it enables structured context propagation: each agent receives only the relevant inputs for its subtask, while outputs from earlier agents can be enriched with retrieved or validation results. Third, agentic workflows support corrective execution: if an agent detects inconsistencies or failures in prior outputs, the workflow can re-invoke earlier agents to revise decisions or recover from errors. This modular design promotes targeted decision-making, minimizes error propagation, and ensures that vulnerability injection proceeds in a systematic, verifiable, and context-aware manner. These properties make agentic workflows well suited to complex code transformation tasks such as vulnerability injection.

III-B Design Methodology and Iterative Refinement

The design of AVIATOR’s workflow was guided by both expert knowledge and iterative refinement. We began by interviewing three experienced cybersecurity researchers (including two co-authors) to document how they would manually inject a vulnerability into secure code. Their reasoning consistently followed four stages: (1) understand the program and its dataflow, (2) identify a suitable injection site, such as sanitization checks that prevent the intended flaw, (3) introduce a subtle but plausible modification at this location, for example by weakening or removing a check, and (4) review the resulting change for plausibility and alignment with the intended CWE. These stages provided the blueprint for AVIATOR’s high-level structure.

Each of these stages was decomposed into elementary subtasks and mapped to specialized agents. For instance, comprehension was assigned to the Code Understanding and Vuln.-Specific Code Understanding agents, and review to a combination of diff-based agents and static analysis. This decomposition reduces the reasoning burden on individual agents and enables the integration of both LLMs and deterministic tools.

The workflow was then refined through repeated experimentation on the SARD test suite 100 (described in §IV-A), which served as our primary development benchmark. Early prototypes highlighted systematic shortcomings: the injector frequently returned truncated code, over-commented edits, or trivial modifications (e.g., whitespace changes). To address these failure modes, we introduced new agents incrementally. Examples include the Polisher, designed to normalize formatting and repair truncated outputs, and the diff-based refinement loop, which detects trivial edits and triggers corrective re-injection.

Crucially, these refinements targeted generic error patterns rather than dataset-specific artifacts. By focusing on issues such as non-substantive edits and truncation, we ensured that improvements generalized across datasets. As shown later in section V, the same workflow behavior transfers to very distinct datasets, confirming that the iterative process captured robust error-recovery patterns rather than overfitting to SARD 100.

Through this iterative design, AVIATOR emerged as a workflow organized into two complementary modules (Figure 1). The Vulnerability Injection Module analyzes benign code, identifies potential injection points, and introduces a vulnerability. The Validation Module then checks these candidates using difference analysis, static analysis, and contextual review, providing structured feedback for correction or reinjection. Together, these two modules instantiate the reasoning process elicited from experts: understanding, modifying, reviewing, and revising.

III-C Vulnerability Injection Module

The Vulnerability Injection Module converts a benign function $c^{b}$ into a candidate vulnerable variant $c^{v}$ of a specified CWE. It instantiates the first three stages of the analyst-inspired workflow: comprehension, vulnerability-specific analysis, and transformation. The module comprises the following agents (Figure 1). We provide the exact prompt templates used by the major agents in AVIATOR in Appendix -A.

III-C1 Code Understanding (LLM)

The process begins with semantic and structural analysis of the benign code. The agent analyzes the control and data flow of the input source code $c^{b}$ to infer its functional intent, input/output behavior, and internal logic. It also qualitatively identifies potential sources—locations where untrusted input may enter the program—and sinks—locations where sensitive operations occur. This stage mirrors the comprehension step a human analyst performs.

III-C2 Information Loader (tool)

Retrieves a detailed definition and description of the type of vulnerability that needs to be injected. This helps ensure the LLM receives proper context on this vulnerability category.

III-C3 Vuln.-Specific Code Understanding (LLM)

Examines the benign code to identify elements that prevent the intended vulnerability from manifesting. These include explicit sanitization checks (e.g., bounds validation) as well as broader safety mechanisms such as correct memory management, error handling, or defensive programming patterns. The agent produces a structured inventory of these safeguards, which downstream agents use to determine how the code can be realistically transformed into a vulnerable variant. This process mirrors traditional symbolic static analysis and taint-tracking techniques [45], but applied inversely. Instead of proving correctness, it highlights the logic that enforces safety. This reverse reasoning is critical to selecting feasible and non-trivial injection points.

III-C4 Vulnerability Injector (LLM + RAG + fine-tuning)

Relies on previous information to apply the transformation and justify its injection. To further boost the performance of this critical component we leverage retrieval-augmented generation (RAG) as well as fine-tuning. In the following, we first explain the retrieval-augmented vulnerability generation, then detail the low-rank adaptation fine-tuning method, followed by supervised fine-tuning and reinforcement learning strategies.

Retrieval-Augmented Vulnerability Generation

Once candidate injection points are identified, an agent transforms the code to inject the target vulnerability (Vulnerability Injector agent). This agent operates under a retrieval-augmented generation (RAG) setup. Given an input function $c^{b}$, the agent retrieves a set of $k$ similar benign/vulnerable pairs $\{c^{b_{(j)}},c^{v_{(j)}}\}_{j=1}^{k}$ from a knowledge base using embedding-based similarity. Retrieved examples are aligned using line-level diffs, marking the injected edits.

This retrieved context alongside the analyzed output, is provided to the LLM to guide code transformation. This ensures that the injection edits reflect realistic code structure and vulnerability patterns observed in real-world examples.

Low-Rank Adaptation (LoRA)

To specialize the vulnerability injection agent without incurring the computational cost of full model fine-tuning, we apply Low-Rank Adaptation (LoRA) [23]. This technique is particularly effective in settings where the base model is large, but the task-specific dataset is limited in size. Rather than updating the full weight matrix $W_{0}\in\mathbb{R}^{d\times k}$, LoRA learns a low-rank decomposition by injecting two trainable matrices $B\in\mathbb{R}^{d\times r}$ and $A\in\mathbb{R}^{r\times k}$, with $r\ll\min(d,k)$, into the model’s architecture: $W=W_{0}+BA$. This reduces the number of trainable parameters while retaining the benefits of large-scale pretraining.

We explore two distinct fine-tuning paradigms for the vulnerability injection agent: supervised fine-tuning (SFT) and reinforcement learning with Group Relative Policy Optimization (GRPO). Both approaches operate on the same dataset of aligned benign/vulnerable function pairs:

[TABLE]

where each benign function $c^{b_{(j)}}$ is accompanied by a structured prompt $q^{(j)}$. The prompt is generated by using the output of prior workflow steps III-C1, III-C2, III-C3. This prompt incorporates (1) semantic and structural analysis results of the benign code, (2) the original benign function itself, and (3) relevant examples from the Retrieval-Augmented Generation (RAG) knowledge base, annotated to highlight vulnerability injection locations. The objective is for the agent to generate the corresponding vulnerable function $\hat{c}^{v}_{j}$ given this prompt.

Supervised Fine-Tuning (SFT)

The vulnerability injection agent is first trained via supervised fine-tuning (SFT), using a standard autoregressive next-token prediction objective. Let $q^{(j)}$ denote the structured prompt. For example $c^{b_{(i)}}$ and $\hat{\mathbf{c}}^{{v}_{i}}=\langle y^{(i)}_{1},\dots,y^{(i)}_{T^{(i)}}\rangle$ the tokenized target vulnerable function of size $T^{(i)}$. We fine-tune the LoRA adapter parameters $\phi$ by minimizing the negative log-likelihood of each next token under teacher forcing. The loss for a batch of size $B$ is:

[TABLE]

Here, $P(\cdot\mid\cdot;\phi)$ is the autoregressive distribution defined by the LoRA-adapted LLM. This technique focuses on exact token-level matching with the aim that it will learn underlying concepts behind vulnerability injection, guided explicitly by ground-truth examples.

Group Relative Policy Optimization (GRPO)

While SFT optimizes for token-level accuracy, it does not directly focus on structural and semantic nuances of code. Therefore, we also apply reinforcement learning (RL) via Group Relative Policy Optimization (GRPO) [47], using the CodeBLEU [43] metric as a reward function.

GRPO is an efficient reinforcement learning technique that eliminates the need for a separate critic network by comparing multiple candidate outputs for each prompt and computing their relative rewards within a sampled group. Given a prompt $q$, we sample $G$ candidate outputs $\{o_{1},o_{2},\dots,o_{G}\}$ from the old policy model $\pi_{\theta_{\text{old}}}$. Each candidate output $o_{i}$ receives a reward score $r(o_{i})$ based on the CodeBLEU metric, which better captures structural and semantic properties of code. CodeBLEU combines four components:

• •

N-gram match: standard BLEU score over tokens.

• •

Weighted n-gram match: gives higher importance to syntax-critical tokens.

• •

AST match: measures similarity of abstract syntax trees.

• •

Data-flow match: compares data-flow graphs to assess semantic equivalence.

[TABLE]

with $\alpha+\beta+\gamma+\delta=1$. By default, $\alpha=\beta=\gamma=\delta$ [43].

The normalized advantage for each candidate output is computed as:

[TABLE]

GRPO optimizes the policy model $\pi_{\theta}$ by maximizing the following clipped surrogate objective:

[TABLE]

where $r_{i}=\frac{\pi_{\theta}(o_{i}\mid q)}{\pi_{\theta_{\text{old}}}(o_{i}\mid q)}$, $\epsilon$ controls clipping for stable optimization, and $\beta$ regulates the strength of the KL-divergence penalty with respect to a reference policy $\pi_{\text{ref}}$. Note that the expectation $\mathbb{E}[\cdot]$ is taken over the random variables $q$ and $\{o_{i}\}_{i=1}^{G}$, with $q\sim P(Q)$ and $\{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(O\mid q)$ and that the KL-divergence penalty encourages policy updates that remain close to the pretrained model:

[TABLE]

III-C5 Polisher (LLM)

After a first version of the injection is completed, this agent was introduced to fix common issues from the Vulnerability Injector. It helps to normalize formatting, corrects truncated outputs, and removes superficial edits such as comment-only changes. The polished candidate is then handed to the Validation Module.

III-D Validation Module

The Validation Module ensures that a candidate $c^{v}$ produced by the Injection Module meaningfully introduces the intended vulnerability into the original function $c^{b}$. It mirrors the review stage of human analysts: checking plausibility, and revising when necessary. The module combines difference analysis, contextual LLM judgment, static analysis, and bounded correction loops (Figure 1).

III-D1 Difference (tool): Triviality Gate

Given $(c^{b},c^{v})$, compute a line diff. If there are no changes or the changes are only whitespace/comments, the injection failed. In that case, the workflow is routed to the Vuln. Injector Identical Code. Otherwise, if there is at least one non-trivial edit, the candidate proceeds directly to the Diff Refiner.

III-D2 Vuln. Injector Identical Code (LLM)

Invoked only when the triviality gate detects no or trivial edits only. This agent re-attempt to generate $c^{v}$ from scratch, incorporating the context provided by prior agents, with the aim of increasing the likelihood that the agent actually introduces the vulnerability this time. The new candidate is returned to the Difference (Triviality Gate) for re-evaluation. This loop is bounded (default: 10 attempts) to prevent inifinite cycling.

III-D3 Diff Refiner (LLM)

For candidates with substantive edits, the Diff Refiner analyzes the generated code together with the annotated difference and determines whether the changes correctly implement the requested vulnerability. If the edits are incomplete, incorrect, or misaligned, the agent updates the vulnerable code accordingly. Its judgment draws on the context accumulated from all previous agents, ensuring that the revised $c^{v}$ both preserves original functionality and aligns with the CWE intent before proceeding to downstream validation.

III-D4 Difference (tool): Reviewer Diff

The revised $c^{v}$ is diffed again to produce the reviewer-oriented, annotated difference consumed by following analyzers.

III-D5 Diff Analyzer (LLM)

The Diff Analyzer serves as a second reviewer to double-check the edits introduced in $c^{v}$. It analyzes the generated code together with the annotated difference and reassesses whether the modifications correctly introduce the requested vulnerability while preserving the original functionality.

III-D6 Lightweight Static Analyzer (tool)

Runs lightweight static analysis on $c^{v}$ using cppcheck111https://cppcheck.sourceforge.io/. Diagnostics are passed to the verifier. While we use cppcheck for C/C++, this component is generic and can be swapped with a static analyzer appropriate for other languages.

III-D7 Vulnerability Verifier (LLM)

Integrates diagnostics from the static analyzer with the annotated diff and CWE context to decide whether the candidate indeed reflects the intended flaw. On success, the module outputs the validated $c^{v}$; otherwise, control transfers to the fixer.

III-D8 Vulnerability Fixer (LLM) and Controlled Restart

Attempts a constrained repair using all accumulated context (code, diffs, CWE, rationales, diagnostics). The repaired candidate re-enters validation at the Lightweight Static Analyzer agent up to one time. If the intended vulnerability still does not materialize, the controller restarts from the initial Vulnerability Injector, exploiting LLM stochasticity to explore a different edit trajectory.To ensure termination, we cap the total number of correction attempts (default: 10) across reinjection and refinement loops.

This hybrid validation strategy combines programmatic and rule-based tools, with LLM-based contextual reasoning, providing robust and interpretable guarantees over the correctness of injected vulnerabilities.

IV Experimental Setup

In this section, we detail the experimental design used to evaluate AVIATOR. We present our research questions, the datasets utilized for both injection validation and downstream tasks, and the implementation details of our framework. We aim to answer the following research questions:

• •

RQ1: How does AVIATOR perform compared to existing automated methods for creating realistic, large-scale vulnerability datasets?

• •

RQ2: What is the contribution of fine-tuning the vulnerability injection agent to AVIATOR’s overall performance?

• •

RQ3: How do different compositions of agents (from monolithic to full agentic workflow) affect AVIATOR’s vulnerability generation capabilities?

• •

RQ4: How does AVIATOR’s injection success rate varies across different vulnerability categories (CWEs)?

• •

RQ5: Can AVIATOR-augmented datasets improve the performance of state-of-the-art deep learning-based vulnerability detection models compared to existing augmentation techniques?

IV-A Datasets

We evaluate AVIATOR specifically for C/C++ functions, given the inherent susceptibility of these languages to low-level vulnerabilities (e.g., memory errors), focusing on the function level in alignment with prior vulnerability detection and repair studies [15]. For RQ1 to RQ4 we used three complementary popular open-source datasets used in previous studies [22] [48] [12]. For RQ5, we additionally used [56] [17] [12] following the methodology established in VulScribeR [10]. Table I summarizes the statistics of each dataset in our experiments.

Entropy

We report an embedding-based entropy score as a proxy for code diversity and richness, complementary to CWE count (which captures label-space coverage). Following [10], we embed each function with CodeBERT, reduce embeddings to three dimensions using principal component analysis, discretize the space into a 10-bin histogram, and compute Shannon entropy over the resulting histogram. Higher entropy indicates a broader variety of code structures and idioms in the dataset, whereas lower entropy reflects more homogeneous or template-like code.

• •

SARD test suite 100 [34, 22]: Contains 102 compact, synthetic test cases, including 34 vulnerable/benign function pairs that can be detected through formal verification tools such as ESBMC [20]. Their compact and self-contained design, makes them ideal for rapid, fully automated validation and served as our primary development benchmark.

• •

FormAI [48]: Includes more realistic and structurally complex synthetic examples. Since it does not provide aligned benign/vulnerable pairs, we manually created benign versions for 37 randomly sampled vulnerable functions, without altering the program’s intent and ensuring no vulnerabilities were detectable by formal verification tools on the resulting benign versions. ESBMC is again used for automated validation, offering a stronger challenge to our injection framework.

• •

PrimeVul [12]: A large dataset of more complex real-world code samples. Due to the lack of full context (e.g., missing class definitions, global variables) and larger code size, formal verification tools like ESBMC are not directly applicable. We use: (1) the paired training split with 3,789 corresponding pairs across 112 CWEs to fine-tune the Vulnerability Injector, (2) a subset of the full training set to train downstream DLVD, (3) the validation split of 480 corresponding pairs (63 CWEs) as the retrieval knowledge base in AVIATOR’s RAG setup, (4) 45 randomly sampled pairs from the test split (24 CWEs) for manual evaluation of injection success, and (5) the full test split (71 CWEs) to evaluate downstream DLVD performance.

• •

Devign [56]: Following previous studies [36, 37, 10], we use Devign as one of the two primary training datasets for the downstream tasks in RQ5. It offers a relatively balanced distribution between vulnerable and clean samples, making it suitable for training robust detection models without excessive class imbalance issues.

• •

BigVul [17]: A large-scale real-world dataset. Consistent with the methodology in VulScribeR [10], we utilize the testing set of BigVul as an independent test set to evaluate the generalization capability of the vulnerability detection models trained on AVIATOR-augmented data (RQ5).

IV-B Implementation Details

Agents

AVIATOR leverages various technologies and tools. Within the validation workflow, we utilized the static analysis tool Cppcheck version 2.17, due to its popularity and ease of integration.

We experimented with two base large language models for the AI agents in our workflow. Qwen2.5-Coder-32B-Instruct [24], is a specialized open-source coding LLM with 32 billion parameters. We additionally experimented with Llama-4-Maverick [32], a general-purpose, mixture-of-experts-based LLM with approximately 400 billion parameters (17 billion active parameters), to explore the impact of model size and specialization on the agents’ reasoning capabilities.

Retrieval-Augmented Generation

We use gte-Qwen2-1.5B-Instruct as the embedding model for vulnerable/benign code function pairs drawn from the validation split of PrimeVul (§IV-A) that constitutes the RAG knowledge base. At the time of our experiments, this model achieved state-of-the-art scores on the MTEB benchmark for embeddings up to 8K tokens [16]. Since 99% of functions in the PrimeVul training split fall below this length, no chunking was required. We set the retrieval parameter to $k=4$, following the recommended default.

The Vulnerability Description used by the Information Loader agent (§III-C2) covers 133 CWE categories and explains common programming mistakes that lead to each vulnerability type.

Fine-Tuning Configurations

We fine-tune the vulnerability injection agent using 3,789 aligned pairs across 112 CWEs from the PrimeVul training set. For supervised fine-tuning (SFT), we trained for 5 epochs (loss convergence) with a learning rate of $2\times 10^{-4}$, batch size of 4, and LoRA ranks varying between 16 and 256. For reinforcement learning with GRPO, we trained the model for 1 epoch (reward convergence), using a learning rate of $2\times 10^{-4}$, batch size of 8, and generating 4 completions per policy iteration. The KL regularization coefficient $\beta$ (0.04) and clipping parameter $\epsilon$ (0.2) followed the recommended default values from GRPO implementation using Adam optimizer.

Technology

AVIATOR is implemented using Python as well as HuggingFace’s Transformers and Parameter-Efficient Fine-Tuning libraries, and VLLM for inference. All experiments were run on NVIDIA A100 or H100 80GB GPUs.

IV-C Empirical Study Design

We conduct an empirical study to quantify the contribution of adding agentic components to the AVIATOR workflow. The goal is to assess both the necessity and the impact of the modular structure by systematically simplifying the framework and comparing outcomes. To this end, we construct and evaluate six progressively reduced variants of AVIATOR, obtained by incrementally removing agents from the full pipeline.

The complete AVIATOR pipeline presented in Figure 1, showcases 13 distinct agents. These agents include tool-based nodes, LLM agents, and control logic designed to manage iterative processes.

The evaluated configurations are as follows:

• •

1-Agent (Monolithic LLM), $W_{1}$: A single LLM receives the original function, vulnerability type, and retrieved examples, and is prompted to analyze the function, inject the vulnerability, and self-verify. This setting relies on common chain-of-thought prompting without agentic decomposition.

• •

3-Agent, $W_{3}$: Decomposes the task into three distinct agents: (i) loading the vulnerability description, (ii) analyzing the function for injection points, and (iii) performing the injection. No verification or refinement stages are included.

• •

5-Agent (Injection-Only Module), $W_{5}$: Implements the full Injection Module (Code Understanding, Information Loader, Vuln.-Specific Code Understanding, Vulnerability Injector, Polisher) without validation (Figure 1, top).

• •

7-Agent, $W_{7}$: Adds the difference-checking loop, consisting of the Diff Agent and Diff Analyzer, allowing the system to detect and iteratively refine the code if no substantive change was introduced.

• •

9-Agent, $W_{9}$: Adds the static analysis tool and corresponding Verifier Agent to further validate the generated vulnerability and refine injection quality based on external tool feedback.

• •

11-Agent, $W_{11}$: Incorporates the Diff Refiner agent after the difference is computed, allowing targeted edits based on diff feedback. This step improves alignment between injection goals and actual edits by making minimal context-aware adjustments.

• •

13-Agent (Full AVIATOR), $W_{13}$: Adds a terminal Vulnerability Fixer Agent. This final pass ensures that injected vulnerabilities are realistic, contextually valid, and non-trivially removable—closely mimicking the post-injection review performed by security experts.

All configurations share the same injection model, prompt formatting, and RAG setup. We evaluate these variants on the same benchmark split (see Section IV-A) and report both formal and manual success metrics (see Section IV-D). This incremental comparison enables a fine-grained understanding of how individual agents affect the correctness, realism, and efficiency of the injection process.

IV-D Evaluation and Metrics

We assess the success rate of vulnerability injection, as it directly indicates the expected label accuracy of datasets produced by our framework.

Automated Evaluation

For compatible test cases [34] and [48], we automatically measured vulnerability injection success using the formal verification tool ESBMC v7.5 (Efficient SMT-based Bounded Model Checker), recognized for its effectiveness in detecting vulnerabilities in bounded code contexts [20, 2]. It implements state-of-the-art incremental Bounded Model Checking and $k$-induction proof-rule algorithms. Namely, if the solver reaches termination within a bound less than or equal to $k$, we can definitively prove the absence of software errors. We set the bound to 100 and timeout to 5 minutes.

Due to the probabilistic nature of LLMs used in AVIATOR, identical inputs may yield different outputs on repeated trials. Therefore, we generated 5 outputs for SARD100 and FormAI test cases. To rigorously evaluate AVIATOR’s performance, we employed two metrics:

• •

Average Injection Success Rate (AISR): Measures the proportion of generated vulnerabilities correctly injected and verified, averaged across multiple runs. Denote $s_{i}$ as the success rate in run $i$ over $n$ runs:

[TABLE]

where $\sigma$ denotes the empirical standard deviation.

• •

Pass@k [6]: Estimates the probability that at least one of $k$ independently sampled outputs is correct. If $c$ is the number of correct outputs among $k$ samples:

[TABLE]

This metric captures the practical success likelihood under multi-sample inference.

These metrics provide insights into AVIATOR’s reliability and practical utility, reflecting its robustness and stability in automated vulnerability injection tasks.

Manual Evaluation

For PrimeVul, where formal verification tools such as ESBMC are not directly applicable due to the lack of full context (e.g., missing class definitions and global variables), as well as bigger code samples, we manually validated a sample of 45 injected functions drawn uniformly at random, following prior work [54, 7, 38, 50]. It is important to note that our total evaluation sample size is consistent with prior similar studies [10, 38, 54]. The evaluation was performed by three authors experienced in C/C++, including two senior security researchers. Each injected sample was categorized as (1) weakness present, (2) no weakness, or (3) unknown, typically due to reliance on external functions or context beyond the injected function. Since we lack complete test cases for most programs, we assess the presence of weaknesses rather than confirmed vulnerabilities (which require demonstrable exploitability). The reported injection success rate reflects the percentage of samples found to contain a weakness, based only on those that could be conclusively assessed.

IV-E Downstream Deep Learning-Based Vulnerability Detection (DLVD) Setup

To answer RQ5, we evaluate the practical utility of AVIATOR by using this workflow to augment training data for downstream vulnerability detection tasks. We adopt the experimental protocol established in recent state-of-the-art studies [10], ensuring a fair comparison against existing augmentation techniques.

IV-E1 Datasets and Augmentation Strategy

We utilized two widely-used benchmarks as seed datasets: Devign [56] and PrimeVul [12], described in Section IV-A. For each dataset, we generated 5,000 new vulnerable samples using AVIATOR and applied specific augmentation strategies to handle class balance, consistently with the methodology in [10]:

• •

Devign Augmentation: We added the 5,000 AVIATOR-generated vulnerable samples to the training set. To prevent label shift and maintain the original class distribution, we simultaneously added a proportional number of benign samples. We sourced these benign samples from [36], which have been previously deduplicated across datasets.

• •

PrimeVul Augmentation: Due to the extreme class imbalance in PrimeVul, we first down-sampled the benign samples in the training set to reduce the ratio from 1:35 to 1:15, making model training possible. After injecting 5,000 AVIATOR-generated vulnerable instances, we applied Random Oversampling (ROS) exclusively to the original vulnerable samples to achieve a balanced 1:1 class ratio for training, to conduct a fair comparison between the methods [10].

For both datasets, we preserved the original distributions of the test sets to ensure that the evaluation assesses performance against real-world distributions.

IV-E2 Models

We evaluated performance across four distinct deep learning architectures, representing two major families of vulnerability detection models:

• •

Graph-based Models: We employed Devign [56] and Reveal [5], which utilize Graph Neural Networks (GNNs) on Code Property Graphs (CPGs) to capture structural semantics.

• •

Transformer-based Models: We employed LineVul [19], a BERT-based model optimized for line-level detection, and CodeT5 [49], an encoder-decoder model pre-trained on code, used as a strong baseline in recent benchmarks [12].

Training Configuration: To ensure a rigorous comparison, we adhered to the training settings reported in [10]. For the graph-based models (Devign, Reveal), we trained each model 5 times with different random seeds and reported the results from the run achieving the highest F1-score. For Transformer-based models (LineVul, CodeT5), we trained for 10 epochs and selected the checkpoint that achieved the highest F1-score.

IV-E3 Baselines

We benchmarked AVIATOR against four distinct baselines:

No Augmentation (No Aug): Training on the original dataset, to establish a reference. 2. 2.

Random Oversampling (ROS): A standard class-balancing technique that duplicates vulnerable samples to mitigate imbalance, shown to be a competitive baseline in prior studies [52]. 3. 3.

VulGen [37] and VGX [36]: State-of-the-art injection techniques that rely on pattern mining and deep learning-based localization. 4. 4.

VulScribeR [10]: The current state-of-the-art downstream-optimized generative augmentation method, leveraging LLMs and retrieval-augmented generation (RAG). VulScribeR explicitly targets downstream DLVD performance rather than injection quality, in contrast to AVIATOR.

V Results and Analysis

In the following we investigate the answers to the research questions posed earlier.

V-A RQ1. Effectiveness of AVIATOR Compared to Existing Dataset Creation Techniques

To evaluate AVIATOR’s effectiveness, we assessed its vulnerability injection success using samples from three benchmarks: SARD100, FormAI, and PrimeVul. Table II shows the average success rates for the full AVIATOR workflow ($W_{13}$) with the SFT injection agent.

AVIATOR achieved a 95% success rate on SARD100 and 91% on the more complex FormAI benchmark, indicating strong generalization beyond simple patterns. For PrimeVul, manual analysis was conducted on 45 injected samples due to the lack of external context needed for automated verification. Out of 34 analyzable cases, 32 contained a verifiable weakness within the function body (94%). The remaining samples involved undefined external symbols, so the analysis focused on confirming the presence of weakness insertion rather than evaluating full vulnerability exploitability.

Table III compares AVIATOR with other techniques, showing that its success rate across all three evaluation benchmarks (93%) surpasses prior methods. Earlier injection methods such as VulGen and VinJ that combine pattern mining with learned localization, report an injection success of 69%, implying that a non-negligible fraction of generated samples do not instantiate the intended weakness. VGX follows a similar pattern-mining paradigm and substantially improves injection success (90%), yet does so under tighter constraints, limited CWE coverage (23 types) and a primary focus on single-statement edits, which restricts the space of vulnerabilities it can realistically generate. In parallel, VulScribeR leverages LLMs and RAG to optimize downstream DLVD performance. However, it does not explicitly prioritize injection correctness and reports a lower estimated injection success (82%). In contrast, AVIATOR’s guided, multi-step injection workflow enables precise, context-aware edits that more closely resemble developer-introduced vulnerabilities, allowing it to remain effective even in partially specified or real-world code settings.

Answer to RQ1: AVIATOR achieves high injection success across benchmarks: 95% on Sard100, 91% on FormAI, and 94% verifiable weakness insertion on PrimeVul. These results highlight AVIATOR’s strong generalization and realism, even in partially specified or real-world code settings. We attribute this performance to AVIATOR’s novel use of an agentic workflow that incorporates multi-step guiding and task decomposition, that enables the injection agent to iteratively refine and validate its own output, likely contributing to its effectiveness across diverse benchmarks.

V-B RQ2. Impact of Fine-Tuning the Injection Agent on AVIATOR’s Performance

To assess the effect of model-level adaptation, we evaluate AVIATOR’s injection performance before and after fine-tuning the injection agent. We experiment with two fine-tuning regimes: supervised fine-tuning (SFT) and reinforcement learning via Grouped Relative Policy Optimization (GRPO), using the same base model (Qwen2.5-Coder-32B) and the full agentic pipeline ($W_{13}$).

Crucially, all fine-tuning is performed exclusively on PrimeVul—a dataset that shares no overlap with either SARD100 or FormAI. This ensures the evaluation measures true generalization, not memorization.

SFT is both more effective and more efficient than GRPO

Table IV reports the average injection success rate ($AISR_{5}$) and its standard deviation across 5 vulnerability injection runs on the same data. On the more complex FormAI dataset, SFT yields significant gains over the base model (91% vs. 85%), while also reducing variance. In contrast, GRPO not only fails to outperform the base model on FormAI (dropping to 84%), but also provides no gain on SARD100. A plausible explanation is that, in this setting, GRPO may be less well-matched to the available training signal and reward formulation, making it harder to consistently reinforce high-quality injection behaviors. Overall, these results indicate that SFT provides a more stable and effective optimization strategy for vulnerability injection in AVIATOR.

On the simpler SARD100 benchmark, the gains from SFT are marginal (95% vs. 94%), indicating that AVIATOR’s base configuration already performs near-saturation. However, the slight improvement and reduced variance still confirm the benefit of tuning.

Pass@k Improvements with SFT

We also reported Pass@k performance to assess the likelihood of producing a successful injection within $k$ generation attempts. Table V shows that SFT improves results consistently across all $k$ on both benchmarks. On FormAI, Pass@1 improves by over 5 percentage points (from 84.3% to 89.9%), and SARD100 reaches near-perfect performance even for small $k$. These gains further validate the robustness added by SFT.

It is important to note that, even without SFT, AVIATOR achieves strong baseline performance. This indicates that the agentic workflow, through decomposition, correction loops, and validation, is sufficient to generate realistic and diverse vulnerabilities. Fine-tuning provides further gains but is not required for the framework to operate effectively. While baseline performance depends on how well the underlying LLM has been trained on the target language, this property remains especially valuable when extending to programming languages or vulnerability classes where aligned training data is scarce, since AVIATOR can still produce rich vulnerability patterns.

Learning dynamics

Figure 2, left, shows the SFT training loss, which demonstrates stable convergence within five epochs. In contrast, GRPO (Figure 2, right) achieved rapid early gains in reward, reaching a high CodeBLEU score by step 400. While the reward plateaus thereafter, it remains stable, indicating that GRPO quickly finds a strong policy and maintains it. However, this comes at significantly higher computational cost, requiring 10 days of training compared to only 10 hours for SFT.

Answer to RQ2. Fine-tuning the vulnerability injection agent using SFT leads to consistent and significant performance gains for the whole AVIATOR workflow, particularly on complex datasets like FormAI. GRPO fails to match this improvement despite being substantially more resource intensive. These results highlight the efficiency and generalization strength of SFT, even when trained on an entirely separate dataset. Additionally, the results show strong baseline performance without fine-tuning, indicating that the agentic workflow alone can inject vulnerabilities. This is essential for transfer to new programming languages or CWE categories where aligned data is unavailable.

V-C RQ3. Contribution of Agentic Components and Model Variants to Injection Performance

To evaluate the contribution of AVIATOR’s agentic workflow design, we conducted an empirical study on seven progressively extended configurations ($W_{1}$–$W_{13}$), incrementally adding specialized agents for analysis, validation, and refinement. We additionally compare four model variants: (i) base Qwen2.5-Coder, (ii) Qwen2.5 fine-tuned using SFT, (iii) Qwen2.5 fine-tuned using GRPO, and (iv) the general-purpose Llama-4-Maverick. Figure 3 shows the average injection success rate across 5 trials ($AISR_{5}$) on both SARD100 and FormAI.

Performance Improves with Workflow Depth

Across all models, performance increases consistently as more specialized agents are added. For example, with the base Qwen2.5-Coder, success rises from 31% at $W_{1}$ to 85% at $W_{13}$ on FormAI, and from 84% to 94% on SARD100. The benefits of modular reasoning are particularly pronounced on complex tasks (FormAI), confirming that decomposing the task into subtasks handled by agents enhances robustness and clarity.

Impact of Specific Agentic Additions

Moving from $W_{1}\rightarrow W_{3}$ introduces basic task decomposition, yielding large but unstable gains on FormAI (+40%) due to better prompt structuring. At $W_{5}$, AVIATOR includes the complete injection pipeline, stabilizing outputs and significantly improving accuracy (+5–10%). Furthermore, by adding feedback-based validation via diff checking ($W_{7}$) and static analysis with cppcheck ($W_{9}$) further boosts performance, particularly for models with more specialized abilities like the fine-tuned Qwen2.5-Coder. Final refiners ($W_{11}$, $W_{13}$) provide consistent final gains across models, enabling subtle bug integration and polishing.

SFT and GRPO Validated Across Configurations

To ensure consistent comparisons, we fine-tuned the vulnerability injection agent starting from $W_{5}$, where the agent’s input structure and prompt format remain unchanged across workflows. Configurations below $W_{5}$ ($W_{1}$, $W_{3}$) are not compatible with our fine-tuned models, as they would require new training due to different agent prompts and inputs. Consistent with results in RQ2, SFT delivers the highest gains across all configurations, while GRPO fails to improve over the base model despite significantly longer training time.

General-Purpose vs. Specialized Models

Llama-4-Maverick, a 400B-parameter generalist model, underperforms all Qwen2.5-Coder 32B variants across the board. For example, on FormAI at $W_{13}$, it reaches 77% compared to 85% with the base Qwen2.5 and 91% with SFT. This underlines the advantage of coding-specialized models and domain-specific fine-tuning for structured tasks like vulnerability injection.

Answer to RQ3. Injection performance improves with workflow refinement: from 31% at $W_{1}$ to 91% at $W_{13}$ on FormAI. Fine-tuning the injection agent using SFT amplifies these gains even though most of the base performance comes from a well-crafted workflow. GRPO fails to outperform the base model. Domain-specialized models (e.g., Qwen2.5-Coder) clearly outperform general-purpose LLMs (e.g., Llama-4), demonstrating the value of both specialization and expert-based agentic decomposition.

V-D RQ4. Performance Variability Across Vulnerability Categories

To further characterize the capabilities of AVIATOR, we analyze injection success rates across the CWE categories present in our evaluation benchmarks (SARD100, FormAI, and the PrimeVul subset). As shown in Figure 4, this distribution reflects only the CWEs present in the datasets; the current AVIATOR instantiation supports 133 CWE types in total as tested in V-F.

Across the evaluated benchmarks, AVIATOR achieves a high overall injection success rate of 93%. In particular, the framework performs strongly on vulnerability types listed in the MITRE Top 25 Most Dangerous Software Weaknesses. This suggests that AVIATOR is well-aligned with real-world security priorities and effectively injects vulnerabilities that matter most in practice.

At a finer granularity, AVIATOR performs well on both simple and complex vulnerability injections. It reliably injects vulnerabilities realizable through localized edits (e.g., missing checks or unsafe memory accesses), while also succeeding on categories that require non-trivial reasoning over program state and execution order. For instance, AVIATOR achieves high success rates on CWE-416 (Use After Free, $N=11$), CWE-476 (NULL Pointer Dereference, $N=22$), and CWE-120 (Buffer Copy without Checking Size of Input, $N=12$), which require consistent reasoning about data flow, pointer validity, and memory access constraints. Beyond memory safety, AVIATOR also performs reliably on arithmetic and control-dependent vulnerabilities such as CWE-369 (Divide by Zero, $N=5$), demonstrating versatility across vulnerability classes.

However, figure 4 also reveals performance variability primarily on vulnerability categories that either lack precise injection semantics or require constraint satisfaction beyond the function scope. For instance, AVIATOR fails to inject CWE-391 (Unchecked Error Condition), which yields a 0% success rate. This CWE lacks a precise modern definition and is slated for deprecation, making it difficult to formulate a concrete injection objective. Similarly, CWE-190 (Integer Overflow) poses a greater challenge, achieving a 67% success rate, because successful injections often depend on satisfying strict arithmetic constraints that involve global assumptions or external inputs. These cases reflect inherent limitations of function-scoped injections rather than shortcomings of AVIATOR’s workflow.

Answer to RQ4: AVIATOR achieves a high overall injection success rate of 93% and performs strongly on practically relevant vulnerabilities, including multiple CWEs from the MITRE Top 25. The framework successfully injects non-trivial vulnerabilities that require reasoning about data flow, control flow, and object lifetimes (e.g., CWE-416, CWE-476), while lower success rates remain confined to ambiguous CWEs or those requiring precise arithmetic reasoning beyond function-level context.

V-E Compute and Cost Estimates

On 8 NVIDIA A100 GPUs, running 128 jobs in parallel, we generated 5,000 vulnerable samples in 4h37m. This averages to 3s per injection with concurrency. At Maryland’s commercial electricity rate (14.82¢/kWh), this corresponds to a cost of about $0.44 per 1,000 injections (GPU power only). These numbers show that AVIATOR can effectively scale dataset generation at very low cost.

V-F RQ5. Impact on Downstream Vulnerability Detection

To assess AVIATOR’s practical utility, we evaluate how augmenting training data with AVIATOR-generated vulnerable samples impacts downstream deep learning-based vulnerability detection (DLVD) performance. Table VI reports the performance metrics for four distinct deep learning architectures (Devign, Reveal, LineVul, CodeT5) trained on the Devign and PrimeVul datasets augmented with generated samples using different strategies.

Consistent downstream gains and comparison to augmentation baselines

Table VI shows that AVIATOR consistently improves downstream DLVD performance across detector families (GNN-based Devign/Reveal and Transformer-based LineVul/CodeT5) and across evaluation settings (cross-dataset Devign$\rightarrow$BigVul and in-domain PrimeVul). Across the eight model-dataset configurations, AVIATOR achieves the best F1 score in 6/8 cases. As further summarized in Table VII, AVIATOR also delivers the largest average F1 improvement among all augmentation strategies. This consistent performance across diverse architectures and datasets indicates that the data generated by AVIATOR exhibits strong generalizability and semantic richness, rather than being tailored to a specific detector family.

AVIATOR leads to an average F1 score improvement of +21.8% relative to the original dataset (No Aug), confirming that the injected samples provide substantial additional training signal beyond the original data. Compared to the Random Oversampling baseline (ROS), AVIATOR achieves a +12.5% average F1 score improvement. Notably, ROS itself outperforms VulGen and VGX by 9.0% and 8.5% in average F1, respectively, highlighting the strength of AVIATOR beyond simple class-balancing effects.

When compared to pattern-based injection methods, AVIATOR improves average F1 score by +25.2% over VulGen and by +24.6% over VGX. Among all baselines, VGX is the only method that achieves a higher average recall than AVIATOR; however, this comes at the cost of substantially lower precision. This behavior aligns with VGX’s reliance on single-statement injection templates and its limited coverage of vulnerability classes (23 CWEs). Such augmentation strategies can make detectors overly sensitive, yielding high recall but low precision (many false positives) which ultimately limits their practical utility for large-scale data augmentation [10]. In contrast, AVIATOR improves F1 through a more balanced precision-recall profile, indicating that it provides a cleaner and less artifact-driven training signal.

Finally, AVIATOR outperforms VulScribeR by +2.9% in average F1, with gains driven primarily by a substantial improvement in recall (+7.4%) while maintaining, and even slightly improving, precision (+1.4%). This precision-recall behavior indicates that AVIATOR’s augmentation introduces less label noise and achieves broader coverage of vulnerable patterns than VulScribeR. In particular, AVIATOR provides detectors with more learnable and discriminative vulnerable signals, enabling them to identify a wider set of true vulnerabilities (reflected by higher recall) without incurring additional false alarms (as evidenced by stable precision). Given VulScribeR’s reported injection success rate of approximately 82%, the observed precision-recall profile is consistent with AVIATOR generating higher-fidelity vulnerable samples on average, with an estimated success rate of 93%, thereby strengthening downstream generalization across detector families.

Overall, these findings suggest that the data generated by AVIATOR is of consistently higher quality than existing augmentation baselines, as reflected by its stable and robust downstream performance across models and evaluation settings.

Distributional fidelity and realism

To further assess the realism of AVIATOR-generated data and explain its downstream gains, we analyzed the distributional properties of the augmented datasets. Figure 5 reports the Kullback-Leibler (KL) divergence between the original training data and the augmented data, using two views: sample counts and entropy distributions across CWE categories. A lower KL divergence indicates that the augmentation strategy preserves the natural shape and diversity distribution of real-world datasets, rather than introducing artificial distortion.

AVIATOR achieves the lowest KL divergence under both metrics, which confirms that it remains closest to the original distribution. VulGen and VGX introduce notably larger shifts, consistent with augmentation that concentrates on a narrower set of injection patterns. VulScribeR reduces this distortion compared to pattern-mining baselines but still deviates more than AVIATOR. Overall, these distributional results align with the precision-recall trends in Table VII: lower distortion corresponds to more realistic, learnable injected vulnerabilities that improve generalization without inducing excessive false alarms.

Operational superiority: quality, cost, and efficiency

Beyond downstream metrics, AVIATOR demonstrates significant operational advantages over the state-of-the-art VulScribeR in terms of data quality and cost-efficiency.

• •

High Syntactic Validity: We utilized the fuzzy C parser Joern [51] to verify the validity of the augmented samples, following the methodology in [10]. While VulScribeR reports a filtration rate (samples rejected due to syntax errors) of 2%–13%, AVIATOR achieves a significantly lower rejection rate: 1.13% for Devign and 1.98% for PrimeVul. This confirms that AVIATOR’s agentic workflow produces more usable training samples with less formatting and parsing noise.

• •

Cost Efficiency: AVIATOR achieves these results at a fraction of the cost. While VulScribeR relies on GPT-3.5 Turbo ($1.88 per 1k samples), AVIATOR’s optimized open-weight workflow costs only **$0.44 per 1k samples**. This represents a 4.3$\times$ cost reduction, making it a far more scalable solution for dataset generation.

Answer to RQ5: AVIATOR delivers the strongest overall augmentation effect in our downstream evaluation. It improves average F1 score by +21.8% over no augmentation and by +12.5% over ROS, and it outperforms all injection baselines, including the prior state-of-the-art LLM-based method VulScribeR (+2.9% average F1). The simultaneous gains in precision and recall, together with the smallest distributional distortion from the original real-world dataset (lowest KL divergence), indicate that AVIATOR generates more realistic and lower-noise vulnerable samples that generalize better across downstream detectors. AVIATOR achieves this quality efficiently, with a $\mathbf{<2\%}$ syntax rejection rate and a 4.3$\times$ lower cost than VulScribeR.

VI Discussion

This section reflects on why AVIATOR is effective, what its results imply for vulnerability dataset construction and downstream detection, and the main limitations of the current study.

VI-A Why Does AVIATOR Work?

We identify the core design choices that explain AVIATOR’s effectiveness in vulnerability injection. These include its agentic decomposition, targeted model-level adaptation and iterative refinement capabilities, mirroring how human experts inject vulnerabilities in practice.

(1) Agentic decomposition provides reliable guidance and controllability. AVIATOR decomposes vulnerability injection into 13 specialized agents, each responsible for a well-defined subtask such as code understanding, localization, transformation, diff-based checking, and refinement. This explicit separation enables structured reasoning and targeted self-correction instead of relying on a single monolithic generation step. Empirically, the workflow design drives most of the gains: injection success on FormAI increases from 31% (single-step baseline) to 91%, and on SARD100 from 84% to 95%. These results indicate that decomposition and tool-augmented verification improve injection fidelity.

(2) Targeted fine-tuning strengthens the core transformation step. Instead of retraining the full pipeline, we fine-tune only the Vulnerability Injector with LoRA-based SFT. This lightweight adaptation (under 10 hours on a single A100 GPU) improves FormAI from 85% to 91% while reducing variance. The model is trained on PrimeVul, a dataset entirely disjoint from the test sets, showing that this adaptation generalizes well across domains.

(3) AVIATOR leverages iterative workflow refinement to improve injection robustness. Agents operate in feedback loops to enhance injection robustness. AVIATOR explicitly detects and corrects common failure modes in LLM code editing (e.g., cosmetic edits, truncation, or insufficient modifications). For example, the Diff Agent can trigger additional refinement passes when changes are overly disruptive or insufficient. This mechanism improves robustness by avoiding brittle, single-statement edits and contributes to consistent performance even without fine-tuning, provided the base LLM has reasonable language coverage. As a result, AVIATOR can adapt to new vulnerability classes and languages, while maintaining diversity in injected flaws.

(4) High-quality injection translates into downstream generalization. KL-divergence analysis shows that AVIATOR induces the smallest shift in CWE sample-count and entropy distributions relative to the original training set. In addition to the high injection success rate, this result supports the downstream precision-recall profile: methods that over-concentrate augmentation on narrow templates tend to distort the training distribution and encourage detectors to produce too many false positives. AVIATOR instead adds signal while tracking the baseline distribution more closely, which plausibly explains its stronger and more stable F1 score gains across detector families.

(5) Operational scalability strengthens practical impact. AVIATOR maintains a low syntax rejection rate under Joern-based filtering ($<2\%$) while reducing generation cost by $4.3\times$ compared to VulScribeR. These properties enable large-scale dataset construction and augmentation without accumulating unusable samples or incurring prohibitive cost, and they make AVIATOR suitable for repeated regeneration under consistent quality controls.

VI-B Limitations and Threats to Validity

Benchmark Scope and Generalizability

Our evaluation focuses on C/C++ using three datasets (SARD100, FormAI, and PrimeVul), which may not capture all real-world diversity. To mitigate this, we combined synthetic and real-world benchmarks of varying complexity. Extending AVIATOR to multi-file contexts and additional languages remains future work.

Function-Level Context Limitations

Our current instantiation operates and validates injections at the function level. For datasets such as PrimeVul, missing project context (e.g., type definitions, globals, external callees) can make some injected samples not determinable under function-local analysis. Accordingly, our manual evaluation assesses whether the intended weakness is present within the function body and marks cases that depend on unavailable context as unknown, as done in prior studies [40]. Ongoing work extends AVIATOR toward repository-level injection and validation.

Manual Evaluation Bias

The qualitative analysis of PrimeVul is based on 45 manually annotated samples, which may introduce subjectivity. We mitigated this by involving three security researchers and resolving disagreements via consensus.

Metric Stability Under Sampling

LLM generation is stochastic, which may affect reliability of metrics like injection success rate. To address this, we performed multiple independent runs and reported average scores $AISR_{k}$ with standard deviations across all experiments and measured Pass@k with additional runs on the best configurations.

Generalization of Fine-Tuned Agent

Our supervised fine-tuning was performed solely on PrimeVul, raising a potential risk of overfitting to its code characteristics. However, PrimeVul includes a wide variety of real-world vulnerabilities across multiple domains. Moreover, the consistent performance improvements on structurally distinct benchmarks (SARD100 and FormAI) provided strong evidence of generalization beyond the training distribution.

Distribution metrics and interpretation

Entropy- and KL-based analyses capture distributional distortion and code diversity proxies, not semantic equivalence to real vulnerabilities. These metrics support our downstream conclusions, but they do not replace direct correctness validation. We therefore interpret them as complementary evidence rather than as ground-truth realism measures.

Cost estimates and hardware dependence

Our cost comparison focuses on generation-time compute under a specific hardware and parallelization setting. Absolute costs vary with deployment, batching, and model choice. The key point is the relative trend: AVIATOR achieves strong quality while enabling lower-cost scaling than large proprietary API-based LLM augmentation pipelines.

VII CONCLUSION

In this paper, we introduced AVIATOR, an AI agentic framework designed to systematically inject realistic and category-specific vulnerabilities into secure codebases. By leveraging a multi-agent pipeline that replicates the reasoning processes of cybersecurity experts, AVIATOR ensures that vulnerability injection is not only precise but also deeply contextually aware. This approach allows for a nuanced understanding and handling of vulnerabilities, closely mimicking the analytical depth and precision of human experts.

Across three widely used benchmarks (SARD100, FormAI, and PrimeVul), AVIATOR achieves high injection success rate (93% across benchmarks), outperforming prior injection techniques and providing stronger guarantees for constructing benchmark datasets with reliable labels. Beyond injection correctness, AVIATOR also delivers the strongest downstream benefits under the standard DLVD augmentation protocol: injecting 5k samples improves average downstream F1 score by +21.8% over no augmentation and +24.6% over VGX, while surpassing the prior downstream state of the art VulScribeR by +2.9% average F1 and +7.4% recall without degrading precision. AVIATOR induces the lowest distributional distortion relative to the original training dataset, supporting the conclusion that it generates more realistic and lower-noise vulnerable samples that generalize across detector families.

AVIATOR is also broadly applicable: our current instantiation supports 133 CWE types, exceeding the coverage of prior pattern-mining approaches (e.g., VGX with 23 CWEs), highlighting strong diversity potential for dataset construction. Finally, AVIATOR is practical at scale, maintaining a $\mathbf{<2\%}$ syntax rejection rate (Joern-based filtering) and achieving a 4.3$\times$ lower generation cost than comparable LLM-based injection pipelines.

Overall, these results show that AVIATOR advances the state of the art in automated vulnerability injection by jointly improving injection fidelity, downstream utility, and scalability. Future work will extend AVIATOR beyond function-level edits toward multi-function and project-level injection, enabling validation under richer context and supporting a broader class of real-world vulnerability scenarios.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Data Availability

Data will be made available upon request.

Declaration of generative AI and AI-assisted technologies in the manuscript preparation process

During the preparation of this work the author(s) used Gemini in order to order to receive suggestions on wording and form of certain paragraphs. After using this tool/service, the author(s) reviewed and edited the content as needed and take(s) full responsibility for the content of the published article.

CRediT authorship contribution statement

Amine Lbath: Writing - Original Draft, Writing – review & editing, Conceptualization, Methodology, Software, Data Curation, Investigation, Visualization. Massih-Reza Amini: Writing - Original Draft, Writing – review & editing, Conceptualization, Methodology, Supervision. Aurelien Delaitre: Writing – review & editing, Conceptualization, Methodology, Supervision, Validation. Vadim Okun: Writing – review & editing, Conceptualization, Methodology, Supervision, Validation.

-A Prompt Templates Used in AVIATOR

This appendix documents the exact prompt templates used by the AI agents in the AVIATOR workflow. Each prompt enforces a strict JSON schema to ensure reliable inter-agent communication, deterministic parsing, and robust validation across iterative refinement loops. All prompts are fixed and reused across experiments.

All prompts utilize the same system prompt: You are a cybersecurity and code analysis expert with deep knowledge in vulnerability detection, secure coding practices, and static code analysis.

-A1 Code Understanding Agent

Prompt Template: Code Understanding Agent

Your task is to analyze the benign function below and try to better understand its goals and its code structure.

{benign_code}

Return your response as a valid JSON object with exactly the following parameters, maintaining correct types:

•

"function_purpose": (str) Provide a proper description of what the code above is intended to do.

•

"input_descriptions": (list[dict[str, str]]) List descriptions explaining all input arguments and what they are used for. Return a JSON list of dictionaries, for example: ["input name": "A string used for ..."].

•

"output_descriptions": (list[dict[str, str]]) List descriptions explaining all return values.

•

"source_sink_list": (list[str]) Exhaustive list of pairs of corresponding code sources and sinks. A source is a data-entry point (e.g., user input), and a sink is a function that should not be called with unsanitized, untrusted data. One source can have multiple sinks and vice versa, but list them only by pairs. Example: [["name of source 1", "name of sink 1"], ["name of source 2", "name of sink 2"]].

•

"data_flows": (list[str]) Analyze and describe all the steps of how data propagates between sources and sinks and throughout the whole code. Return a JSON list, where each line is a textual description of all the steps the data follows between one source and one sink. Do this also throughout the entire code.

Do not repeat yourself.

Ensure to write the JSON object in between these markers: ‘‘‘json and ‘‘‘

-A2 Vuln.-Specific Code Understanding Agent

Prompt Template: Vuln.-Specific Code Understanding Agent

Your task is to identify sanitization and vulnerability mitigation elements that makes the code below benign.

The benign source code to analyze is:

{benign_code}

The purpose of this code is described as: {function_purpose}

Its inputs are: {input_descriptions}

Its inputs are: {output_descriptions}

Use the following list of sources / sinks and dataflow as starting point to identify the different ways the vulnerability described in vul_inject_info is avoided in this benign source code:

•

"source_sink_list": {source_sink_list}

•

"data_flows": {data_flows}

The vulnerability that is prevented in this code can be described as follows:

{vul_inject_info}

Return your response as a valid JSON object with exactly the following parameters, maintaining correct types:

•

"sanitization": (list[dict[str, str]]) Identify and analyze all sanitization and vulnerability mitigation elements that make the benign code actually non-vulnerable in terms of the vulnerability described above. Return a JSON list of dictionaries [{"description": "Actual explanation here", "code": "corresponding code here"}]

Important:

•

Do not include any additional text or formatting outside the JSON object.

•

Ensure the JSON strictly adheres to the specified parameter names and types.

•

Ensure to write the JSON object in between these markers ```json

{ "sanitization": [ {"description": "Actual explanation here", "code": "corresponding code here"}, ... ] }

and ```

•

Be sure to close every string you open with "

•

Do not ever repeat yourself.

Now, please generate the JSON assessment.

-A3 Vulnerability Injector Agent

Prompt Template: Vulnerability Injector Agent

Your task is to modify the source code of non-vulnerable function below to introduce the requested vulnerability.

The non-vulnerable code that must be modified to introduce changes that will make this code vulnerable is:

{benign_code}

The purpose of this code is described as: {function_purpose}

Its data flows are: {data_flows}

The list of sanitization and vulnerability mitigation elements that made the benign code actually non-vulnerable are:

{sanitization}

Think step by step and leverage the previous analysis of the code to change the code to introduce this vulnerability: {vul_inject_info}

As a model you can leverage these examples of how a benign code was modified to introduce the vulnerability {vul_inject_id}:

{retrieved_context}

Return your response as a valid JSON object with exactly the following parameters, maintaining correct types:

•

"vulnerable_code": (str) Modified source code with vulnerability introduced. Write your code in between these tags <raw> full vulnerable code </raw>. Rewrite the full code, do not truncate by adding comments such as "//... rest of the code...". Do not add comments indicating where the vulnerability was introduced.

Important:

•

Do not include any additional text or formatting outside the JSON object.

•

Ensure the JSON strictly adheres to the specified parameter names and types.

•

Ensure to write the JSON object in between these markers ‘‘‘json and ‘‘‘

•

Ensure to write the code in between these tags <raw> full vulnerable code </raw>

•

Write the code for the full function with modifications where the original benign code is changed to introduce a new vulnerability.

•

Output only a valid JSON object. Do not include markdown formatting (no triple backticks or extra quotes). Please ensure that any internal quotes are escaped properly.

•

Do not ever repeat yourself.

Now, please modify the benign function accordingly and output only the JSON object as specified. Be sure to finish the JSON with ‘‘‘

-A4 Polisher Agent

Prompt Template: Polisher Agent

You are tasked to verify that the benign code below has been properly modified to introduce the CWE vulnerability {vul_inject_id}.

The initial non-vulnerable source code function was:

The new version of the code that must have been modified to introduce changes that supposedly make this code vulnerable:

Verify that the code was properly modified to introduce the following vulnerability: {vul_inject_info}. You must ask yourself the following questions and think step by step as an expert:

•

You claimed that introducing the following changes would transform the code to become vulnerable. Does this really introduces the CWE vulnerability {vul_inject_id} and actually modifies the code?

•

If yes, did you actually made the change? Are you sure?

•

If no, can you modify the initial non-vulnerable source code function to introduce the requested vulnerability.

•

Either way, verify that the vulnerability was introduced in a realistic way, as if a human developer had made a mistake.

•

Make sure the code still makes sense and did not lose its functionality.

•

Have you truncated the code to only show the modifications? The vulnerable code should show the full code.

If you need to make some modifications, rewrite the values. Otherwise, if everything is correct, return the values without changes.Return your response as one valid JSON object with exactly the following parameters, maintaining correct types:

•

"vulnerable_code": (str) Modified source code with vulnerability introduced. Write your code in between these tags <raw> full vulnerable code </raw>. Make sure you checked the questions above and the code is properly vulnerable.

•

"injection_location": (str) Location where you have modified the code to introduce the vulnerability (line number, function section and explanation of the location).

•

"injection_justification": (str) Detailed explanation of why you have modified the code to introduce this vulnerability in this specific way, referencing function analysis and retrieved context.

•

"confidence_score": (int) Score between 0 and 10 on how confident you are that your answer is correct and the vulnerable code actually is vulnerable with the proper vulnerability.

•

"confidence_justification": (str) Justification of the confidence score.

Important:

•

Do not include any additional text or formatting outside the JSON object.

•

Ensure the JSON strictly adheres to the specified parameter names and types. Do not write the types.

•

Ensure to write the JSON object in between these markers ‘‘‘json and ‘‘‘.

•

Output only a valid JSON object. Do not include markdown formatting (no triple backticks or extra quotes). Please ensure that any internal quotes are escaped properly.

•

Do not ever repeat yourself.

Now, please make the verifications and output only the JSON object as specified.

-A5 Diff Refiner Agent

Prompt Template: Diff Refiner Agent

You are tasked with verifying that the changes made to the code below actually correspond to the requested vulnerability and that the generated vulnerable code is not missing any parts from the original code that would change its functionality.

The original benign code is:

{benign_code}

The vulnerable code that was produced is:

{vulnerable_code}

The vulnerability to introduce is:

vul_inject_id: vul_inject_info

The purpose of this code is described as: function_purpose

Its data flows are: data_flows

The list of sanitization and vulnerability mitigation elements that made the benign code non-vulnerable are:

{sanitization}

Your tasks:

Analyze the differences between the benign and vulnerable code.

Ensure that the changes made actually introduce the requested vulnerability.

Ensure that the vulnerable code is not missing any parts from the original code that would change its intended functionality.

If the changes are insufficient or incorrect, update the vulnerable code to properly introduce the vulnerability while preserving the original functionality.

Return your response as a valid JSON object with exactly the following parameters, maintaining correct types:

•

"diff_analysis": (str) Analysis of whether the code changes correspond to the requested vulnerability and if any functionality is lost.

•

"injection_location": (str) Location where you have modified the code to introduce the vulnerability (line number, function section and explanation of the location).

•

"injection_justification": (str) Detailed explanation of why you have modified the code to introduce this vulnerability in this specific way, referencing function analysis and retrieved context.

•

"confidence_score": (int) Score between 0 and 10 on how confident you are that your answer is correct and the vulnerable code actually is vulnerable with the proper vulnerability.

•

"confidence_justification": (str) Justification of the confidence score.

Important:

•

Do not include any additional text or formatting outside the JSON object.

•

Ensure the JSON strictly adheres to the specified parameter names and types.

•

Ensure to write the JSON object in between these markers json and .

•

Ensure to write the code in between these tags <raw> full vulnerable code </raw>.

•

Output only a valid JSON object. Do not include markdown formatting (no triple backticks or extra quotes). Please ensure that any internal quotes are escaped properly.

•

Do not ever repeat yourself.

Now, please analyze and, if needed, update the vulnerable code and output only the JSON object as specified. Be sure to finish the JSON with ```.

-A6 Diff Analyzer Agent

Prompt Template: Diff Analyzer

You need to critically analyze the code changes made to introduce a vulnerability and verify that they properly implement the intended vulnerability type.

The original benign code was:

{benign_code}

The vulnerable code that was produced is:

{vulnerable_code}

The difference between the benign and vulnerable code is shown below with [ADDED] and [REMOVED] markers:

{code_diff}

The vulnerability that should have been introduced is:

{vul_inject_id}: {vul_inject_info}

You are claiming to have made the following modification:

•

Location: {injection_location}

•

Justification: {injection_justification}

Your task is to critically analyze whether:

The modifications actually match what was claimed in the injection_location

The introduced changes actually correspond to the vulnerability described in vul_inject_info

The vulnerability has been properly implemented, not just superficially simulated

The code changes are substantive and not merely cosmetic

Return your response as a valid JSON object with exactly the following parameters, maintaining correct types:

•

analysis_result: (str) Your detailed critical analysis explaining whether the code modification properly introduces the intended vulnerability. Include specific references to the code changes and vulnerability requirements.

•

modification_valid: (bool) A boolean value indicating whether the modification properly introduces the intended vulnerability (true) or if it fails to do so (false).

Important:

•

Do not include any additional text or formatting outside the JSON object.

•

Ensure the JSON strictly adheres to the specified parameter names and types.

•

Ensure to write the JSON object in between these markers ‘‘‘json and ‘‘‘

•

Be critical and thorough in your assessment - the goal is to verify that the vulnerability was properly introduced.

•

Do not ever repeat yourself.

Now, please analyze the code changes and provide your assessment as specified.

-A7 Vulnerability Verifier Agent

Prompt Template: Vulnerability Verifier Agent

Role. You are a comprehensive vulnerability verification system with advanced knowledge of common programming flaws, security issues, and code vulnerabilities.

Goal. Perform a final validation of whether the intended vulnerability has been properly introduced in the code, using all available evidence.

Inputs.

•

Original benign code: {benign_code}

•

Produced vulnerable code: {vulnerable_code}

•

Intended vulnerability: {vul_inject_id}: {vul_inject_info}

•

Prior assessments:

–

Critical analysis result: {analysis_result}

–

Modification validity: {modification_valid}

•

Static analysis signals:

–

CWE IDs detected: {cwe_ids}

–

Static analysis errors: {static_analysis_errors}

•

Diff evidence: {code_diff} (with [ADDED] / [REMOVED] markers)

•

Claimed modification:

–

Location: {injection_location}

–

Justification: {injection_justification}

Your tasks (must follow all).

Evaluate whether the CWE detection matches the intended vulnerability type.

Cross-reference the code diff with the static analysis findings.

Determine whether the vulnerable code exhibits the weakness described in {vul_inject_info}.

Verify that the modification is realistic and meaningful (not cosmetic).

Confirm that the intended vulnerable pattern has been implemented correctly.

Output format. Return one valid JSON object with exactly the following parameters (maintaining correct types):

•

"is_correctly_vulnerable": (bool) Whether the code correctly implements the intended vulnerability.

•

"verification_explanation": (str) Detailed explanation with evidence from static analysis, diff, and prior assessments.

Important:

•

Do not include any additional text or formatting outside the JSON object.

•

Ensure the JSON strictly adheres to the specified parameter names and types.

•

Ensure to write the JSON object in between these markers ‘‘‘json and ‘‘‘.

•

Be comprehensive and thorough in your final assessment.

•

Do not ever repeat yourself.

Now, please provide your final verification of the vulnerability introduction as specified.