AEGIS : Automated Co-Evolutionary Framework for Guarding Prompt Injections Schema

TL;DR

AEGIS introduces an automated co-evolutionary framework that enhances the robustness of LLMs against prompt injection attacks through iterative attack and defense prompt optimization.

Contribution

It presents a novel gradient-like prompt optimization technique enabling autonomous evolution of attack and defense prompts against prompt injections.

Findings

Achieves attack success rate of 1.0, outperforming baselines.

Improves detection true positive rate by 0.23, reaching 0.84.

Effective across different LLMs and validated on real-world datasets.

Abstract

Prompt injection attacks pose a significant challenge to the safe deployment of Large Language Models (LLMs) in real-world applications. While prompt-based detection offers a lightweight and interpretable defense strategy, its effectiveness has been hindered by the need for manual prompt engineering. To address this issue, we propose AEGIS , an Automated co-Evolutionary framework for Guarding prompt Injections Schema. Both attack and defense prompts are iteratively optimized against each other using a gradient-like natural language prompt optimization technique. This framework enables both attackers and defenders to autonomously evolve via a Textual Gradient Optimization (TGO) module, leveraging feedback from an LLM-guided evaluation loop. We evaluate our system on a real-world assignment grading dataset of prompt injection attacks and demonstrate that our method consistently outperforms existing baselines, achieving superior robustness in both attack success and detection. Specifically, the attack success rate (ASR) reaches 1.0, representing an improvement of 0.26 over the baseline. For detection, the true positive rate (TPR) improves by 0.23 compared to the previous best work, reaching 0.84, and the true negative rate (TNR) remains comparable at 0.89. Ablation studies confirm the importance of co-evolution, gradient buffering, and multi-objective optimization. We also confirm that this framework is effective in different LLMs. Our results highlight the promise of adversarial training as a scalable and effective approach for guarding prompt injections.

Equations

$$\Delta S_{\text{rel}}=\{\begin{array}{ll}\frac{S_{\text{attacked}}-S_{\text{benign}}}{S_{\text{max}}-S_{\text{benign}}}& \text{if }{S}_{\text{benign}}<S_{\text{max}}\\ 0& \text{if }{S}_{\text{benign}}\ge S_{\text{max}}\end{array}$$

$$S_{\text{attack}}=w_{\text{asr}}\cdot (\text{ASR}{)}^{p_{\text{asr}}}+w_{\text{sc}}\cdot (\Delta S_{\text{rel}}{)}^{p_{\text{sc}}}$$

$$S_{defense}=w_{tp}\cdot (TPR{)}^{p_{tp}}+w_{tn}\cdot (TNR{)}^{p_{tn}}$$

Full text

AEGIS : Automated Co-Evolutionary Framework for Guarding Prompt Injection

** Ting-Chun Liu*†*   Ching-Yu Hsu*†*   Kuan-Yi Lee*†*   Chi-An Fu*†*   Hung-yi Lee**

Electrical Engineering, National Taiwan University

{b10901039,b10901036,b10901091,b11901174}@ntu.edu.tw  hungyilee@ntu.edu.tw

\text{\textdagger}\text{\textdagger}footnotetext: Equally contribution.

1 Introduction

Large Language Models (LLMs) have rapidly become integral components of modern AI systems, powering a wide range of downstream applications such as education. However, the deployment of LLMs in real-world settings also exposes them to security risks, most notably prompt injection attacks, where maliciously crafted inputs manipulate the model into producing unintended or harmful outputs. Unlike traditional adversarial examples, prompt injections exploit the semantic and contextual flexibility of natural language, making them particularly challenging to detect and defend.

Existing defense mechanisms largely fall into two categories: training-based approaches that require additional fine-tuning of LLMs, and training-free approaches that rely on manually designed prompts, templates, or heuristics. While the latter are attractive for their efficiency and compatibility with black-box LLMs, they often suffer from limited robustness and adaptability because of their dependence on fixed, human-engineered designs. Recent work in prompt optimization has demonstrated the potential of systematic search strategies to improve prompts for performance, generalization, or safety. Yet, most optimization frameworks assume static objectives, leaving open the question of how to adapt defenses in adversarial and evolving environments such as prompt injections.

In this work, we introduce AEGIS , a novel adversarial co-evolution framework for automated discovery of robust defenses against prompt injection attacks, as illustrated in Figure 1. Our framework jointly evolves attack and defense prompts in an iterative, GAN-inspired process, where attackers continuously refine adversarial strategies and defenders adapt in response. The core of AEGIS is TGO+, an enhanced textual gradient optimization module that simulates gradient-like updates using natural language feedback. By leveraging multi-route optimization signals, a gradient buffer, and adversarial co-training, AEGIS autonomously explores the space of defensive strategies without requiring model fine-tuning or human-crafted rules.

We evaluate our framework on automated assignment grading, a realistic scenario where malicious prompts can manipulate grading outcomes. Across multiple LLMs, AEGIS consistently improves both attack and defense prompts over successive iterations, achieving state-of-the-art robustness against real-world injection attempts while preserving utility on benign inputs. Furthermore, ablation studies confirm the critical role of co-evolution and gradient replay in sustaining long-term robustness.

Our main contributions are as follow

• •

Proposing a general co-evolutionary adversarial framework that systematically evolves both attackers and defenders, enabling adaptive robustness against prompt injection attacks.

• •

Designing TGO+, an enhanced prompt optimization method with multi-route textual gradients and gradient buffering, tailored for black-box LLMs.

• •

Demonstrating the effectiveness of our framework in a specific real-world application, conducting comprehensive experiments on authentic datasets and multiple LLMs to showcase superior defense performance and strong cross-model generalizability.

• •

Through ablation studies and prompt evolution analysis, we highlight the importance of co-evolutionary training and provide insights into how prompts improve over time.

Together, these contributions establish a principled and automated approach to defending against prompt injections, advancing the reliability and security of LLM-powered applications.

2 Related Works

A key observation we make is that most existing training-free defenses against prompt injection still depend heavily on human-crafted design or heuristic insights. In this section, we highlight three recent yet fundamentally different approaches, each representative of a popular defense strategy, and show how they align with our observation.

2.1 Training-Free Defenses

• •

LLM-based Detection Prompts. PromptArmor Shi et al. (2025) prompts an LLM to identify and remove injected content, but the detection prompt is manually crafted and not optimized for varying threat settings.

• •

Input-level Structure Encoding. Spotlighting Hines et al. (2024) inserts provenance markers to separate user input from system instructions, though the marker format is hand-engineered and static across tasks.

• •

Behavioral Consistency Checking. MELON Zhu et al. (2025) re-executes tasks with a fixed masking prompt to detect indirect injections. While task-agnostic, its masking strategy is still manually specified and may not cover diverse attacker behaviors.

All three methods rely on fixed, manually designed prompts or templates. Our framework instead automates prompt and transformation exploration, enabling adaptive, more robust defenses against diverse injection attacks.

2.2 Prompt Optimization

Our work connects to the broader landscape of prompt optimization. Following the terminology used in Cui et al. (2025), our prompt optimization framework falls under the category of heuristic-based prompt search algorithms. Cui et al. (2025) categorize prompt optimization techniques by their target objectives. To facilitate comparison, we further organize existing objectives into two broad categories, based on whether the optimization setting assumes a static or evolving task environment.

2.2.1 Under Static Environment

This category includes works that optimize prompts for known and fixed objectives, where the task definition and evaluation criteria remain constant throughout.

Task-Specific Optimization aim to improve performance on specific downstream tasks under static conditions, optimizing metrics such as accuracy or BLEU without addressing adversarial dynamics. Pryzant et al. (2023) propose ProTeGi, a heuristic method that refines prompts using natural language feedback called textual gradients, beam search, and bandit-based selection. Chen et al. (2024) introduce PROMST, which integrates rule-based and learned heuristics to optimize instructions and demonstrations across multi-step tasks. Opsahl-Ong et al. (2024) present MIPRO, a black-box meta-optimization framework that jointly refines prompts in multi-stage LM programs using program-aware proposals and surrogate evaluation.

Cross-Domain Optimization, for example, Li et al. (2024) propose Concentrate Attention, which uses attention strength and stability in deeper transformer layers to guide prompt optimization. They introduce a concentration-based loss for soft prompts and a reinforcement learning strategy for hard prompts, achieving better out-of-domain performance without sacrificing in-domain accuracy.

Multi-Objective Optimization, such as Sinha et al. (2024), propose Survival of the Safest (SoS), a multi-objective evolutionary framework that optimizes prompts for both task performance and safety. By interleaving semantic and feedback-based prompt mutations, SoS identifies candidate prompts that balance accuracy and robustness, under a fixed threat model.

2.2.2 Under Evolving Environment

This category includes works that adapt to evolving task demands or adversarial settings. It is suited for evolving task environments, where objectives or threats may evolve over time. This direction has received relatively limited attention due to the complexity of modeling dynamic behaviors.

Robust Prompt Optimization (RPO), proposed by Zhou et al. (2024), shares a similar adversarial optimization framework with ours. However, it adopts a white-box setting, where the defender simulates the attacker’s optimization process using gradient-based methods (e.g., GCG) on open-source models. For closed-source models, the defensive suffixes are optimized on open-source surrogates and then directly transferred for evaluation without further adaptation on the black-box target. In contrast, our method assumes a black-box setting, where attacker and defender are optimized without access to model internals or gradients.

3 Method

We propose a general adversarial co-evolution framework that enables both attackers and defenders to evolve automatically through iterative optimization. Although the framework can be applied to different security-sensitive tasks, we focus on automated assignment grading as a concrete scenario to demonstrate its effectiveness. In this setting, an attacker attempts to obtain high scores by injecting adversarial prompts, and the defender aims to prevent misgrading. By leveraging LLM-guided feedback and prompt refinement, our system continuously improves both attack and defense strategies without human intervention.

The core training procedure is illustrated in Figure 2. In this framework, the attacker and defender evolve in alternating turns. In each cycle, the attacker evolves for a fixed number of iterations based on the current best defense. Once the attacker finishes its training and gets the current best attack, the defender will evolve in response to it. This process continues until both sides converge (i.e., no further improvements) or a predefined maximum number of GAN iterations is reached. The overall algorithm can be seen in Appendix A.1.

We describe different modules in the following sections.

3.1 Attacker and Defenser

3.1.1 Attacker

The attacker module aims to generate adversarial prompts that will be further injected to the original prompt and can mislead the system to give a higher score. Within the co-evolutionary framework, the attacker evolves in alternating turns against a fixed defender, simulating an arms race between offensive and defensive strategies.

During each attacker iteration, a training phase is initiated to explore new adversarial candidates. Specifically, new attack candidates ($ATK\_Cand_{j}^{i}$) are generated using the $TGO^{+}$ module, which synthesizes gradient-like signals derived from grading feedback and guides the editing of existing prompts to enhance their adversarial strength. The attacker will then maintain a top-$k$ pool of the strongest attack prompts from previous cycles, denoted as $ATK^{i}_{j}$, by evaluating their effectiveness against the current best defense using $Eval()$ in the grading system.

To evaluate the effectiveness of each new attack candidate, we ask the LLM in the grading system to output three important values based on the original input and the attack candidate:

• •

$S_{\text{benign}}$: The score assigned to the original (benign) input without any adversarial prompts.

• •

$S_{\text{attacked}}$: The score assigned after injecting adversarial prompts into the original input.

• •

ASR: The attack success rate, which quantifies the probability that an attack bypasses detection by the defense.

After generating these values, we first calculate the relative score change $\Delta S_{\text{rel}}$ in Equation (1):

[TABLE]

where $S_{\text{benign}}$ means the maximum possible score defined by the grading system.

If $S_{\text{benign}}<S_{\text{max}}$, the relative score change is computed as the ratio between the observed improvement due to the attack and the maximum improvement that could possibly be achieved. Conversely, if $S_{\text{benign}}\geq S_{\text{max}}$, no further improvement is feasible, and $\Delta S_{\text{rel}}$ is set to zero. This normalization is critical because it accounts for the diminishing significance of score increments near the upper end of the grading scale. For instance, an increase from 8 to 9 carries greater weight than an increase from 1 to 2, as improvements become progressively harder to obtain as scores approach the maximum.

After computing the relative score change $\Delta S_{\text{rel}}$, we rank and maintain the top-$k$ adversarial attacks using the attack score defined in Equation (2).

[TABLE]

where $w_{asr}$ and $w_{sc}$ are weights for the ASR and $\Delta S_{rel}$, respectively, and $p_{asr}$ and $p_{sc}$ are power parameters to control the sensitivity of each term.

The weighting coefficients ($w$) determine the relative importance of the two metrics in the overall score—for example, assigning a larger $w_{\text{asr}}$ prioritizes attacks that are more difficult for the defense to detect. The power parameters ($p$) modulate how strongly changes at different regions of the metric’s range influence the score—for instance, a larger $p_{\text{asr}}$ amplifies the effect of improvements in high-success regions (e.g., from 0.8 to 0.9) relative to low-success regions (e.g., from 0.1 to 0.2).

After the training procedure, the best-performing attack ($ATK^{i}_{best}$) from the top-$k$ pool is chosen with the highest attack score on the validation set using $Val()$ in the grading system. Then, the selected adversarial prompts will be fixed for the next defender evolution cycle, ensuring that the defender is trained against the most challenging known threat at the time.

By leveraging LLM-based editing, gradient-guided refinement, and evaluation signals from the grading system, the attacker adaptively explores the adversarial prompt space, driving the co-evolutionary process forward.

3.1.2 Defender

The defender module aims to develop prompts that are robust against adversarial attacks and capable of eliciting accurate system responses. Unlike the attacker, which seeks to exploit model weaknesses, the defender focuses on maintaining reliability under adversarial pressure.

Following the co-evolutionary setup, the defender evolves in response to a fixed attacker. After each attacker cycle, the best-performing attack prompt is used as the evaluation context against which the defender is trained. The evolution process mirrors that of the attacker: new defense candidates ($DEF\_Cand^{i}_{j}$) are generated via the $TGO^{+}$ module, and a top-$k$ pool of defense prompts denoted as $DEF^{i}_{j}$ is maintained via $Eval()$.

While the underlying mechanics—generation, evaluation, and selection—are symmetric to the attacker’s process, the defender faces a different optimization goal. Rather than maximizing disruption, the defender is trained to neutralize the attack while preserving the semantic intent and correctness of responses. This often requires precise prompt calibration and semantic grounding, especially in high-stakes settings.

Ultimately, the defender provides a moving target for the attacker, contributing to the dynamic equilibrium of the co-evolutionary training process.

To evaluate the effectiveness of the defense prompt, we ask the LLM in the grading system to output two important values: True Positive Rate (TPR), which denotes the probability that the defense correctly detects an attack, and True Negative Rate (TNR), which denotes the probability that the defense correctly identifies a benign input as non-attacked.

After generating these two values, we will calculate the defense score in Equation (3):

[TABLE]

where $w_{tp}$ and $w_{tn}$ are weights for the True Positive Rate (TPR) and True Negative Rate (TNR), respectively, and $p_{tp}$ and $p_{tn}$ are their corresponding power parameters. Similar to the attack score, the coefficients and power parameters in the defense score allow for a nuanced and flexible evaluation of the effectiveness of the defense prompt.

Similarily, the best defense ($DEF^{i}_{best}$) is chosen from the top-$k$ pool with the highest defense score on the validation set using $Val()$ in the grading system after the training process.

3.2 TGO+ for Prompt Optimization

Inspired by the TGO framework proposed in  Pryzant et al. (2023), we adopt a modular design to optimize prompts through gradient-like updates in natural language space. Each TGO+ module is dedicated to a specific optimization goal (e.g., attack or defense) and operates in two stages: gradient acquisition and gradient application, as illustrated in Figure 3. The overall algorithm for TGO+ can be seen in Algorithm 2.

3.2.1 Gradient Acquisition

For each prompt and its gradient results (ASR, $\Delta S_{\text{rel}}$ for attack prompt; TPR, TNR for defense prompt), the module first collects the errors on these gradient results. These errors are then combined with a task-specific instruction and prompted to LLM, and LLM returns several feedback messages serving as the textual gradients—i.e., suggestions indicating how the prompt could be improved.

To ensure diverse learning, recent feedback messages are stored in a gradient buffer. This buffer encourages diversity in the optimization trajectory by prompting the LLM to generate alternative gradients even when the same input prompt is provided.

3.2.2 Gradient Application

In the second stage, the feedback is synthesized into a set of guidance messages based on the optimization type (e.g., ASR optimization for attack / TPR optimization for defense). These are used to construct an edit prompt that instructs the LLM to revise the original candidate prompt accordingly. After all these prompt candidates are generated, they are sent to $Eval()$ in the grading system to evaluate their effectiveness.

TGO+ enables gradient-like prompt updates without requiring access to model internals, making it compatible with black-box LLMs such as GPT-4o and Gemini-2.5-flash.

3.2.3 Key Innovations

Our implementation of TGO+ introduces several key innovations that differentiate it from the original work:

• •

Multi-Route Gradient Optimization: To enhance the optimization process, we employ a multi-route gradient strategy. This means that for each prompt, we generate textual gradients based on multiple optimization type. For instance, the Attackerś prompts are optimized based on either ASR or the relative score change. Similarly, the Defenderś prompts are optimized based on TPR or TNR. This allows for a more holistic and effective optimization process.

• •

Gradient Buffer: We introduce a gradient buffer that stores past textual gradients. This prevents the model from repeatedly making the same mistakes and encourages the exploration of novel optimization pathways.

4 Experimental Setup

4.1 Dataset

The dataset for our experiment can be separated into two parts: one for training, and one for real-world evaluation. During the training phase, we use a total of 50 GPT-generated benign articles. For these 50 articles, 40 of them are used during training, and 10 of them are used for validation. For the real-world evaluation phase, we use 143 malicious articles collected from student submissions in the previous course at National Taiwan University, which is the same course in Chiang et al. (2024). These articles contain a wide variety of successful prompt injections that achieve full scores without being detected by the defense, which serve as the baseline for calculating the True Positive Rate (TPR). Furthermore, we select another 100 benign articles from the course. These 100 articles do not contain any injections, which serve as the baseline for calculating the True Negative Rate (TNR) of the defense. All student-submitted articles (143 malicious ones + 100 benign ones) have been manually modi- fied to anonymize personal data and for copyright purposes, while preserving their original strategic intent

4.2 Experimental Procedure

To ensure the reliability and stability of our findings, all experiments were conducted three times. The results presented in this paper are the average values from these three runs. We also calculated the standard deviation of these experiments. This statistic calculation mitigates the impact of stochasticity in the training process and provides a more robust measure of performance.

4.3 Hyperparameters

The default hyperparameters used in our experiments are summarized in Appendix A.3. These settings were used for the baseline experiment, and variations are explored in the ablation studies.

5 Results

To comprehensively evaluate our framework, we benchmark its performance against several established baseline methods and analyze its iterative improvement over the training process.

5.1 Overall Evaluation

We evaluate the defense effectiveness of our framework against three baseline mechanisms, including Perplexity-based Detection Alon and Kamfonas (2023), LLaMA 3.1 Guard Inan et al. (2023), and the "Human-Crafted Prompt" defense, which is proposed in Chiang et al. (2024) to defend against real-world attacks (See Appendix A.6). For all these methods, we evalaute the defense against the real-world articles (147 malicious articles + 100 benign articles). The results, summarized in Table 1, show that our method achieves state-of-the-art defense performance. We present our results at both an early stage (Iteration 4) and the final stage (Iteration 8) of the GAN training.

As shown, our defender at Iteration 4 already outperforms the strong LLaMA 3.1 Guard baseline. By Iteration 8, our method keeps improving, achieving the best balance of a high True Positive Rate (0.84) and a high True Negative Rate (0.89), demonstrating its superior ability to identify sophisticated attacks while maintaining utility on benign inputs.

5.2 Iterative Performance

To illustrate the co-evolution of the attacker and defender, Table 2 shows the real-world evaluation for both agents at different iteration of the GAN training process. To be more specific, we evaluate the generated attacks against the defense in Human-Crafted Prompt from Chiang et al. (2024), and evaluate the generated defenses against the real-world articles (143 malicious articles + 100 benign articles) to calculate the TPR and TNR. The results demonstrate a clear trend of mutual improvement, where each agent becomes progressively stronger by adapting to the other.

The trend shows the attacker’s metrics (ASR, Relative Score Change) steadily increasing as it learns to bypass the improving defender. Simultaneously, the defender’s TPR improves as it learns to find the defense prompt to detect the adversarial attacks. This dynamic demonstrates a successful adversarial training loop, leading to highly robust agents.

Appendix A.4 provides examples of real prompt refinements, illustrating that the defense yields meaningful qualitative improvements.

5.3 Cross-Model Generalizability

To evaluate the robustness and generalizability of the framework, we run the experiment on different LLMs.

5.3.1 Framework Generalizability

We first test whether our framework can be transfered on different LLMs, including GPT-5-mini, GPT-4.1-nano, Gemini-2.0-flash, Gemini-2.5-flash-lite. The results are shown in Table 3, We can see that all these models achieve better TPR at higher GAN iteration compared with lower GAN iteration, indicating that this framework can be applied on various LLMs. For more detailed results, please check the Appendix A.5.1.

5.3.2 Prompt Transferability

We also investigated the transferability of the generated defense prompts across various large-language models (LLMs). Specifically, we examine whether the prompts optimized using one model (in this case, GPT-4.1-mini) retain their effectiveness when applied to different LLMs (in this case, GPT-4.1-nano, Gemini-2.5-flash, and Gemini-2.5-flash-lite) without modification.

Table 4 illustrates the results for all models at GAN iteration 0, iteration 4, and iteration 8. For stronger LLMs (GPT-4.1-mini, Gemini-2.5-flash), they achieve a high TPR at iteration 8, while weaker LLMs (GPT-4.1-nano, Gemini-2.5-flash-lite) achieve lower TPR at iteration 8. However, we can see that the results for all LLMs improve when more GAN iterations are trained. For more detailed results, please check the Appendix A.5.2.

6 Ablation Study

To understand the contribution of each component in AEGIS , we conducted several ablation studies. These studies involve systematically removing or altering parts of our system and observing the impact on performance.

6.1 Without Gradient Buffer

We removed the gradient buffer, which stores historical textual gradients for the optimization prompt. This change leads to a slower convergence rate and degrades the TPR in defense about 5%.

6.2 Without Multiple Gradients

We simplify the gradient generation process to use only one optimization type. For the defender, we used only the True Positive Rate (TPR), and for the attacker, only the Attack Success Rate (ASR). This led to a performance degrade in both attacker and defender, where the final performance of the defense degrades with over 10

6.3 Single-Sided Training

We also experimented with training only one side of the GAN framework. The lack of an adaptive adversary meant that the trained model quickly overfit to its static opponent, highlighting the importance of co-evolution. When training the defender against a static set of attacks, it fails to generalize to new, unseen attacks, resulting in a lower overall TPR.

The comparisons between all the ablation studies are presented in the figures above. Fig 5 compares the TPR between each ablation, and Fig 4 compares the TNR between each ablation.

7 Conclusion

We presented AEGIS , a novel automated co-evolutionary framework for guarding against prompt injection attacks in Large Language Models (LLMs). By iteratively optimizing both attack and defense prompts using a gradient-based natural language strategy, AEGIS systematically explores the prompt space without manual engineering. Our approach demonstrates superior performance over baseline and hand-crafted prompts on several LLMs, achieving promising results in both attack strength and defense robustness.

Through extensive experiments and ablation studies, we confirmed the importance of co-evolution, gradient replay, and multi-objective optimization. These findings suggest that adversarial training, when applied at the prompt level, offers a scalable and effective solution for safeguarding LLMs in real-world deployments. In future work, we plan to extend AEGIS to more complex scenarios and improve prompt interpretability.

Limitation

Despite the promising results, our study has several limitations. First, our evaluation focused on automated assignment grading task, which may not fully capture the diversity of real-world security-sensitive tasks and show the generalizability of our framework. Second, our defense method mainly targets text-based dialogue systems, and its effectiveness in multimodal systems remains unclear. Third, while we focused on quantitative evaluation of attack success rates and defense robustness, large-scale human evaluations were not conducted. Addressing these limitations in future work will be important for building more comprehensive and deployable defense systems.

Appendix A Appendix

A.1 Algorithm for Adversarial Co-evolution Framework

Algorithm 1 outlines the overall adversarial co-evolution procedure. In each iteration, the attacker and defender are alternately optimized through a prompt-based generation and evaluation process. Candidates are produced using the prompt optimization framework $TGO^{+}$, and their effectiveness is evaluated using task-specific criteria by $Eval()$. From these candidates, the best-performing attacker and defender are selected at the end of each iteration via $Val()$. This co-evolutionary process continues until the maximum number of iterations $N$ is reached, at which point the framework outputs the final attacker and defender models, $ATK^{N}_{best}$ and $DEF^{N}_{best}$.

A.2 Algorithm for TGO workflow

Algorithm 2 describes how new candidate prompts are generated and refined during the adversarial co-evolution process. For each candidate $c$, the framework first identifies the error cases, and these failures are summarized into a natural language error string, providing contextual feedback. Then, this error string is augmented with task-specific instructions and gradients collected from past iterations to enable experience replay. The LLM will respond with textual gradients from the error string, and the LLM can further use these gradients to generate the new candidate prompt.

A.3 Detailed Hyperparams

Table 5 shows the detailed hyperparameters we use in our experiments:

A.4 Defense Prompt Improvement

This part, we demonstrate the improvement of the detection prompt in the training phase. Table 6 shows the improvement in the defense prompt from the earlier stage to the late stage, and their respective TPR and TNR evaluated on the student-submitted articles (143 malicious ones + 100 benign ones).

In this example, the defense prompt is relatively short and lacks clarity in iteration 0, providing only general instructions without sufficient detail to address complex prompt injection attempts. By iteration 4, the prompt incorporates more specificity, including example scenarios such as the presence of subtle, conditional, or positively framed language that could bias the evaluation. In iteration 8, the prompt becomes longer and even more detailed, explicitly covering a broader range of potential manipulations. These include instructions with indirect appeals, positively framed requests, conditional statements, and phrasing designed to encourage specific scoring outcomes, emphasize only strengths, or disregard weaknesses. This progression reflects a steady improvement in the prompt’s comprehensiveness and precision throughout the GAN training process, and the progression can be seen by the improvement in TPR. The above progression is highlighted in Table 6.

A.5 Detailed Experiment Results

This part presents the complete TNR values for all evaluated large language models (LLMs) across different GAN training iterations.

A.5.1 Cross-Model Generalizability (Full Results)

Table 7 and Table 8 contain the TPR and TNR values for GPT-5-mini, GPT-4.1-nano, Gemini-2.0-flash, and Gemini-2.5-flash-lite at each GAN iteration respectively. We can see that TPR keeps increasing among all the GAN iterations despite a slight degradation in TNR values for all LLMs.

A.5.2 Prompt Transferability (Full Results)

Table 9 and Table 10 show the TPR and TNR values when detection prompts generated by GPT-4.1-mini are transferred to be used on other LLMs without modification respectively. The results show that there is a strong transferability since the detection prompt generated by the source model can still achieve a nice TPR value when transfered to other LLMs.

A.6 Human Crafted Prompt

Chiang et al. (2024) introduces the prompt used to detect whether there are attacks in the students’ articles, and we slightly modified the detection prompt that leads to a higher TPR, which is called the Human Crafted Prompt. The full prompt is in Table 11