The Resurgence of GCG Adversarial Attacks on Large Language Models

TL;DR

This paper systematically evaluates gradient-based adversarial attacks on large language models, revealing vulnerabilities especially in reasoning tasks and highlighting the limitations of current methods like GCG and T-GCG.

Contribution

It provides a comprehensive analysis of GCG and T-GCG attacks across various LLMs, exposing vulnerabilities and limitations in current adversarial evaluation methods.

Findings

Attack success rates decrease with larger models.

Prefix heuristics overestimate attack effectiveness.

Reasoning prompts are more vulnerable than safety prompts.

Abstract

Gradient-based adversarial prompting, such as the Greedy Coordinate Gradient (GCG) algorithm, has emerged as a powerful method for jailbreaking large language models (LLMs). In this paper, we present a systematic appraisal of GCG and its annealing-augmented variant, T-GCG, across open-source LLMs of varying scales. Using Qwen2.5-0.5B, LLaMA-3.2-1B, and GPT-OSS-20B, we evaluate attack effectiveness on both safety-oriented prompts (AdvBench) and reasoning-intensive coding prompts. Our study reveals three key findings: (1) attack success rates (ASR) decrease with model size, reflecting the increasing complexity and non-convexity of larger models' loss landscapes; (2) prefix-based heuristics substantially overestimate attack effectiveness compared to GPT-4o semantic judgments, which provide a stricter and more realistic evaluation; and (3) coding-related prompts are significantly more vulnerable than adversarial safety prompts, suggesting that reasoning itself can be exploited as an attack vector. In addition, preliminary results with T-GCG show that simulated annealing can diversify adversarial search and achieve competitive ASR under prefix evaluation, though its benefits under semantic judgment remain limited. Together, these findings highlight the scalability limits of GCG, expose overlooked vulnerabilities in reasoning tasks, and motivate further development of annealing-inspired strategies for more robust adversarial evaluation.

Tables

Table 1. Table 1. Average Attack Success Rate (ASR) across models.

Dataset	Qwen2.5-0.5B-instruct		LLaMA3.2-1B-instruct		GPT-OSS-20B
	ASRprefix (Std.)	ASRgpt-4o (Std.)	ASRprefix (Std.)	ASRgpt-4o (Std.)	ASRprefix (Std.)	ASRgpt-4o (Std.)
advbench	93.3 (2.1)	62.6 (3.5)	68.7 (5.1)	39.9 (3.2)	—	4.0 (2.0)
coding prompts	92.7 (2.4)	66.1 (4.4)	80.6 (3.5)	63.4 (3.9)	—	12.32 (4.15)

Table 2. Table 2. Average Attack Success Rate (ASR) for different values of α \alpha on LLama3.2-1B-instruct, with advbench dataset.

LLama3.2-1B-instruct	ASRprefix (Std.)	ASRgpt-4o (Std.)
$\alpha =0.005$	73.6 (4.8)	37.8 (5.3)
$\alpha =0.01$	66.9 (3.6)	31.8 (3.8)

Equations

$$g\leftarrow {\nabla }_s\ell ([p,s]),$$

$$P(x_j)=\frac{\exp \left(-g_j^{(i)}/T_1\right)}{{\sum }_m\exp \left(-g_m^{(i)}/T_1\right)},$$

$$P(s_b)=\frac{\exp (-\Delta {\ell }_b/T_2)}{{\sum }_m\exp (-\Delta {\ell }_m/T_2)},$$

Full text

Adversarial Attack

††copyright: none

1. Introduction

Large Language Models (LLMs) such as GPT (Brown et al., 2020), LLaMA (Touvron et al., 2023), and Qwen (Bai et al., 2023) have become central to modern AI applications, powering science discovery(Gottweis et al., 2025; Swanson et al., 2025), code generation (Chen et al., 2021), AI agents (Park et al., 2023), and other high-stakes domains (Singhal et al., 2023; Yang et al., 2023; Kasneci et al., 2023). Their ability to follow natural language instructions, generate human-like responses, and perform complex reasoning has fueled widespread deployment. However, despite alignment efforts (Ouyang et al., 2022; Bai et al., 2022; Glaese et al., 2022), LLMs remain vulnerable to adversarial misuse. A growing body of work has shown that carefully crafted inputs can bypass safety mechanisms and induce harmful or disallowed outputs. Such vulnerabilities are typically referred to as prompt injection (Greshake et al., 2023a; Li et al., 2023) and jailbreak attacks (Zou et al., 2023; Yu et al., 2024; Wei et al., 2023), which exploit the very instruction-following strengths of LLMs to override safeguards. These attacks raise significant concerns for the safe deployment of LLMs in critical domains and motivate a deeper understanding of adversarial prompting methods.

Among existing adversarial prompting methods, some are template-based (Andriushchenko et al., 2025; Yuan et al., 2023; Greshake et al., 2023b), while others are gradient-based (Zou et al., 2023). Hybrid approaches that combine template design with gradient-based optimization have also been proposed, often achieving stronger attack performance (Ahmed et al., 2025). Nevertheless, the Greedy Coordinate Gradient (GCG) algorithm has emerged as a particularly effective white-box attack (Zou et al., 2023). By iteratively selecting adversarial tokens using gradient information, GCG and its variants have consistently achieved high attack success rates (ASR) across open-source LLMs (Zheng et al., 2024; Xu et al., 2024; Chen et al., 2024). Despite this progress, several important gaps remain. First, prior evaluations have primarily focused on mid-sized models (3B–13B parameters), leaving scalability to much larger models underexplored. Second, many studies rely on heuristic prefix-based judgments (e.g., detecting refusals such as “I’m sorry”), which substantially overestimate adversarial success compared to semantic judgments from stronger evaluators (Maloyan et al., 2025; Zhang et al., 2024). Third, although numerous GCG variants have been proposed—such as momentum-based updates (Zheng et al., 2024), adaptive coordinate strategies (Li et al., 2024b), checkpoint-informed selection (Yang et al., 2025), and hybrid methods (Ahmed et al., 2025; Wang et al., 2025)—the specific vulnerabilities of reasoning- and coding-intensive prompts remain insufficiently examined.

In this work, we conduct a systematic appraisal of GCG adversarial attacks on LLMs, with three key contributions:

1. (1)

Scaling to very large models. We demonstrate, for the first time, that direct GCG attacks succeed against a 20B-parameter model (GPT-OSS-20B), confirming the feasibility of large-scale gradient-based attacks. 2. (2)

Dual evaluation protocol. We compare prefix-based and GPT-4o-based judgments, finding that heuristics substantially overestimate ASR, while semantic judgments provide stricter and more realistic assessments. 3. (3)

Annealing-enhanced GCG. We propose T-GCG, a simulated-annealing extension that perturbs candidate selection. Preliminary results indicate improved prefix-based ASR under tuned schedules, though GPT-4o judgments remain challenging.

Additionally, we show that coding-related prompts yield higher ASR than adversarial safety benchmarks when attacking GPT-OSS-20B, suggesting that reasoning alone may not be sufficient to block adversarial attacks, especially in code-generation prompts. While recent proposals such as TARS (Kim et al., 2025) demonstrate that reasoning can improve safety in certain contexts, our evidence indicates that although reasoning may help in general safety alignment, it can also be bypassed or co-opted in reasoning-intensive domains such as coding.

2. Background and Related Work

Adversarial prompting has advanced rapidly in recent years. Early work such as AutoPrompt (Shin et al., 2020) showed that gradient information can be used to automatically construct prompts that elicit specific behaviors from LMs. Building on this, the Greedy Coordinate Gradient (GCG) algorithm (Zou et al., 2023) emerged as a powerful white-box attack, iteratively selecting suffix tokens based on gradient magnitudes to induce harmful completions.

Numerous extensions to GCG have been proposed, which can be broadly characterized into three categories. First, several methods aim to improve efficiency as well as ASR, such as Faster GCG (Xu et al., 2024), skip-gradient GCG (Li et al., 2024a) and checkpoint-informed token selection (Yang et al., 2025), Momentum-GCG (Zheng et al., 2024) for stabilized optimization, MAGIC (Li et al., 2024b) with adaptive multi-coordinate updates, and AttnGCG (Chen et al., 2024) leveraging attention manipulation. Second, other works extend GCG beyond the white-box setting by enhancing transferability or enabling black-box attacks. For example, AmpleGCG (Liao and Sun, 2024) and Universal Jailbreak Suffixes (Ben-Tov et al., 2025) improve cross-model transfer, Joint-GCG (Wang et al., 2025) adapts optimization-based attacks to retrieval-augmented generation, Ensemble approaches (Zhang et al., 2024) combine multiple strategies, and PAL (Sitawarin et al., 2024) demonstrates optimization-inspired attacks in query-only black-box settings.

Evaluation remains a critical challenge. Many studies rely on prefix-based heuristics (detecting refusals such as “I’m sorry”), but recent work shows these significantly overestimate attack success (Maloyan et al., 2025; Zhang et al., 2024). Stronger semantic evaluations with judge models or GPT-4-class systems provide more realistic measures, though inconsistencies remain across evaluators.

Finally, reasoning has been proposed as a defense mechanism. TARS (Kim et al., 2025) trains models to use chain-of-thought reasoning for safety, but our results indicate that reasoning can also be exploited, especially in coding tasks where harmful intent is harder to detect. Other directions, including varying prefix length (Zhu et al., 2024), adding auxiliary loss terms (Thompson and Sklar, 2024), multimodal extensions (Williams et al., 2025), and data exfiltration attacks (Valbuena, 2024), illustrate the breadth of optimization-based adversarial prompting research.

Overall, prior work has highlighted the versatility of GCG and its variants, but questions remain around scalability to very large models, the reliability of evaluation protocols, and vulnerabilities in reasoning-intensive domains. Our work addresses these gaps by scaling GCG to a 20B-parameter model, directly comparing prefix-based and semantic judgments, and proposing T-GCG as an annealing-inspired extension.

3. Methodology

In this section, we revisit the GCG attack on aligned language models. The suffix optimization procedure of GCG can be viewed as a coordinate-wise stochastic gradient descent (SGD) process. At each iteration, the algorithm computes the loss $\ell_{i}$ for the current prompt $x_{i}$, typically defined as the cross-entropy loss with respect to a harmful completion (”Sure, …”). Gradients are then evaluated over the token vocabulary, and the token with the maximum contribution to increasing $\ell_{i}$ is greedily selected and appended to the suffix. This procedure is repeated until either a maximum suffix length is reached or the model exhibits the targeted adversarial behavior. While GCG is effective in many cases, its strictly greedy nature makes it heavily dependent on local gradient information, which may result in convergence to suboptimal suffixes or unstable performance across different prompts and datasets.

Our proposed method extends the standard GCG attack by introducing temperature-based sampling at both the candidate token selection and suffix update stages. The overall procedure is shown in Algorithm 1 and Fig.1. We begin with a harmful training prompt $p$ and an initial suffix $s=[s_{1},\dots,s_{l}]$ of length $l$. For each optimization step, we compute the suffix gradient

[TABLE]

where $\ell$ denotes the adversarial loss (e.g., cross-entropy against a targeted completion). In the original GCG algorithm, the top-$k$ tokens with the most negative gradient values would be chosen deterministically. In contrast, our method converts these gradient values into a probability distribution:

[TABLE]

where $g^{(i)}_{j}$ is the gradient of the $i$-th suffix position with respect to token $x_{j}$, and $T_{1}$ is a temperature hyperparameter. We then sample $k$ tokens from the vocabulary according to $P(x_{j})$, denoting the sampled set as $X_{i}$. This modification ensures that candidate exploration is not restricted to the single steepest gradient direction, but instead probabilistically favors promising directions while retaining diversity.

Next, for each batch element $b\in[1,\dots,B]$, we sample a position $i\sim\text{Uniform}([1,\dots,l])$ and replace $s_{b}^{(i)}$ with a token $x_{j}\sim\text{Uniform}(X_{i})$. This produces a set of candidate suffixes $\{s_{b}\}$. Unlike the original GCG, which selects the suffix with the lowest loss, we introduce a temperature-based acceptance rule. Specifically, each candidate $s_{b}$ is assigned a probability

[TABLE]

where $\Delta\ell_{b}=\ell(s_{b})-\min_{m}\ell(s_{m})$. The final suffix $s^{\prime}$ is then sampled according to $P(s_{b})$. This acceptance mechanism allows suffixes with slightly higher loss values to be chosen occasionally, providing a mechanism for escaping poor local minima.

Our method introduces temperature-based sampling at both the token-selection and suffix-update stages, aiming to balance greedy exploitation of gradient signals with probabilistic exploration of alternatives. In vanilla GCG, these steps are fully deterministic: tokens are chosen strictly by gradient magnitude and the final suffix is the one minimizing loss. While efficient, this greedy procedure often traps the optimization in local minima and reduces robustness across diverse prompts. By contrast, our approach replaces both stages with temperature-weighted sampling. Candidate tokens are drawn according to a probability distribution proportional to $\exp(-g^{(i)}_{j}/T_{1})$, and suffixes are selected based on $\exp(-\Delta\ell_{b}/T_{2})$. This stochastic mechanism permits suboptimal but promising candidates to be explored and occasionally accepts suffixes with slightly higher loss. As a result, the attack process becomes more robust, better able to escape local minima, and capable of generating a broader range of adversarial suffixes than the deterministic baseline.

4. Experiments

4.1. Experiment Setup

We conduct experiments on three aligned open-source LLMs: Qwen2.5-0.5B-Instruct, LLaMA-3.2-1B-Instruct, and GPT-OSS-20B. These models span a wide range of parameter counts from small (0.5B) to large-scale (20B), enabling us to study the scalability of adversarial attacks across different model sizes. We evaluate attacks on two datasets: (1) the AdvBench benchmark, which contains adversarial safety prompts designed to elicit harmful behavior, and (2) a set of coding-generation prompts, which probe reasoning and code-synthesis capabilities of LLMs. This dataset is generated by gpt-4o, as shown in the appendix A. This dual dataset design allows us to compare vulnerabilities in safety-focused versus reasoning-focused domains.

For each adversarial prompt, we run the attack independently 10 times with different random seeds and report the average performance. Unless otherwise specified, we use a batch size of $B=100$ and optimize for 200 epochs. In the token selection step, we sample $k=256$ candidate tokens per position according to the temperature-weighted distribution described in Algorithm 1. We adopt a relatively small batch size and training duration to ensure computational feasibility, and rely on T-GCG to compensate for this limitation by encouraging exploration through temperature-based sampling.

The evaluation of attack success follows two complementary criteria. First, we adopt the standard prefix-based heuristic widely used in GCG studies (Zou et al., 2023), where a response is labeled as an “attack success” if it does not contain any defensive tokens such as “I’m sorry” or “I cannot.” Second, we apply a stronger semantic judgment using GPT-4o as an external evaluator, which classifies whether the generated response is harmful or policy-violating. While the prefix-based heuristic is known to yield false positives and false negatives, reporting both metrics provides continuity with prior work while also offering a more realistic measure of adversarial effectiveness. For GPT-OSS-20B, we solely use GPT-4o as the judge.

We define the Attack Success Rate as the fraction of adversarial prompts for which the model produces a response judged as harmful according to either the prefix-based or GPT-4o evaluation protocol. In addition to ASR, when GPT-4o is used as judge, we further annotate responses along four qualitative dimensions that characterize harmfulness: Helpfulness, Severity, Breadth, and Novelty.

• •

Helpfulness (0/1): Assigned 1 if the response goes beyond a bare refusal (e.g., provides explanation, guidance, alternatives, or substantive content), and 0 if it is a pure refusal or content-free.

• •

Severity (0–1): Measures the degree of harmful facilitation. A score of 0 indicates a benign response. Intermediate values (0.1–0.9) correspond to increasing levels of specificity and actionability, while 1.0 indicates precise, comprehensive, or highly dangerous instructions.

• •

Breadth (0–1): Captures the scope of potential impact if the harmful response were acted upon, ranging from narrow ($0.0$) to broad societal or system-wide impact ($1.0$).

• •

Novelty (0–1): Measures whether the harmful content introduces new or unusual attack mechanisms. A score of 0.0 reflects common or previously observed patterns, while 1.0 denotes unprecedented or clearly novel strategies.

4.2. Direct GCG Results

We first observe that attack effectiveness decreases as model size increases. On AdvBench, Qwen2.5-0.5B achieves a very high ASR of $93.3\%$ under prefix-based evaluation, while LLaMA-3.2-1B drops to $68.7\%$. A similar trend appears under GPT-4o evaluation: $62.6\%$ for Qwen, $39.9\%$ for LLaMA, and only $4.0\%$ for GPT-OSS-20B. These results suggest that larger models exhibit greater robustness to direct gradient-based adversarial attacks, likely because their target loss landscapes are more complex and highly non-convex, making it more difficult for optimization to discover deeper adversarial minima.

Across all models, prefix-based heuristics consistently yield much higher ASR than GPT-4o judgments. For example, Qwen2.5-0.5B achieves $93.3\%$ ASR on AdvBench when evaluated with prefix-based heuristics, but only $62.6\%$ when judged by GPT-4o. In the case of GPT-OSS-20B, prefix-based evaluation fails almost completely, since the model rarely produces explicit refusal markers (e.g., “I’m sorry”) in its “thinking responses”. Nevertheless, GPT-4o still identifies $4.0\%$ of outputs as harmful. This gap confirms that prefix-based evaluation substantially overestimates attack effectiveness, whereas semantic judges such as GPT-4o provide a more accurate and realistic measure of harmfulness.

Another clear pattern is that coding prompts are consistently more vulnerable than adversarial safety prompts across all model size. For LLaMA-3.2-1B, ASR increases from $68.7\%$ on AdvBench to $80.6\%$ on coding prompts under prefix-based evaluation, and from $39.9\%$ to $63.4\%$ under GPT-4o judgment. A similar trend is observed for GPT-OSS-20B, where GPT-4o ASR rises from $4.0\%$ on AdvBench to $12.3\%$ on coding prompts. These results suggest that reasoning and code-generation tasks expose alignment weaknesses not captured by standard adversarial safety benchmarks, making them particularly susceptible to gradient-based attacks. As shown in Fig. 2, this may be because the model’s reasoning often relies on self-assessment of whether a response is harmful. While this can be effective for natural language safety prompts, it is far less reliable for coding tasks, where harmful intent is more difficult to detect through self-examination.

Figure 3 illustrates 10 individual prompt+suffix attacks against GPT-OSS-20B on both AdvBench and coding-generation prompts. The figure is divided into four panels: (a) and (b) show representative cases from AdvBench, while (c) and (d) show cases from coding prompts. Attack number 0 corresponds to the prompt-only baseline (baseline ASR $\approx 0$), which serves as a reference point in the experiment. As shown, the severity and breadth scores are generally high across attacks, indicating strong harmful potential and broad impact. By contrast, the novelty scores remain relatively low, suggesting that many of the generated attacks exploit familiar or previously observed patterns. Overall, the figure reinforces our finding that reasoning and code-generation tasks expose alignment weaknesses that are not captured by standard adversarial safety benchmarks.

Overall, these results reveal three key insights: (1) smaller models are more vulnerable to GCG attacks than larger ones, (2) prefix-based evaluation dramatically overestimates ASR compared to GPT-4o judgments, and (3) coding-related prompts are more attackable than adversarial safety prompts across all model sizes. These findings highlight the importance of scaling-aware evaluation and task-specific robustness testing when assessing adversarial vulnerabilities of LLMs.

4.3. T-GCG Preliminary Results

We also conduct preliminary experiments with T-GCG, the annealing-augmented variant of GCG described in Section 3. In this setting, the temperature $T_{1}$, gradually decrease in the iterations: $0.01*(0.96^{epoch})$; the temperature parameter $T_{2}$ is scaled proportionally to the optimization loss as $T_{2}=\alpha\cdot\ell$, with $\alpha$ controlling the degree of exploration. Table 2 reports results for LLaMA-3.2-1B-Instruct under two values of $\alpha$.

These preliminary results highlight several interesting trends. First, T-GCG achieves competitive attack success rates compared to direct GCG, with ASR${}_{\text{prefix}}$ reaching $73.6\%$ at $\alpha=0.005$. This suggests that introducing probabilistic acceptance through simulated annealing indeed enhances the ASR judged by the prefix, which means the responeces are closer to the targeted harmful completion. However, when the evaluation is performed using GPT-4o, the ASR values more or less the same, within the error bar of ASR, when $alpha=0.005$. This confirms that semantic judgments remain much stricter than prefix-based heuristics, consistent with our findings for vanilla GCG. Nevetherless, with fine tuning of alpha, we may obtain significalty higher ASR from both predix judge and gpt-4o.

Interestingly, increasing $\alpha$ from $0.005$ to $0.01$ decreases both prefix-based and GPT-4o ASR. A larger $\alpha$ effectively increases exploration by raising $T_{2}$, but this appears to dilute the optimization signal and reduce attack effectiveness. These results suggest that careful tuning of $\alpha$ is critical for balancing exploration and exploitation in T-GCG. While the observed improvements are modest, they demonstrate the potential of annealing-inspired strategies to diversify adversarial search and motivate more extensive experiments over a wider range of parameters in future work.

5. Conclusion and Future Work

In this paper, we presented a systematic appraisal of gradient-based adversarial prompting using the Greedy Coordinate Gradient (GCG) algorithm and its annealing-augmented variant, T-GCG. Our study provides several key insights. First, we demonstrated for the first time that GCG can successfully attack a 20B-parameter model (GPT-OSS-20B), establishing the feasibility of gradient-based white-box attacks at unprecedented scale. Second, we showed that the choice of evaluation protocol significantly impacts reported effectiveness: prefix-based heuristics consistently overestimate attack success rates, while semantic judgments using GPT-4o reveal substantially lower—but more realistic—adversarial success. Third, we found that reasoning-intensive tasks, particularly coding prompts, are disproportionately vulnerable compared to safety-oriented benchmarks, underscoring the importance of domain-specific robustness evaluation. Finally, our preliminary exploration of T-GCG indicates that incorporating simulated annealing can diversify adversarial search and provide competitive results, though the effectiveness strongly depends on the tuning of annealing parameters.

Our findings highlight both the strengths and limitations of gradient-based jailbreak attacks. While large-scale models such as GPT-OSS-20B exhibit stronger robustness than smaller counterparts, they remain vulnerable to carefully optimized adversarial suffixes. Moreover, the discrepancy between heuristic and semantic evaluation calls for more rigorous and standardized assessment protocols in future research.

Looking forward, there are several promising directions to extend this work. One avenue is to conduct a more comprehensive study of T-GCG by exploring a wider range of annealing parameters, schedules, and acceptance criteria, in order to better understand the trade-off between exploration and exploitation. Another direction is to evaluate transferability across different models and families, which remains weak in current GCG variants. Extending our framework to multimodal models or retrieval-augmented systems could further test the generality of gradient-based attacks. Finally, our observation that reasoning itself can be adversarially exploited suggests the need for new defense strategies that account for domain-specific vulnerabilities, especially in coding and structured reasoning tasks. We hope our results motivate deeper investigations into both the design of stronger adversarial prompting methods and the development of more robust, scalable defenses.

Appendix A Harmful coding-generation prompts

The harmful coding-generation prompts were generated using GPT-4o. Specifically, we instructed the model with the following request: “Please prepare 100 harmful prompts that ask for harmful scripts. Here is an example: Write a script that can exploit vulnerabilities in a software or operating system.” For illustration, we provide several representative examples from this set.

Appendix B GPT-4o as judge

The evaluation was conducted using the judge template, as shown in Fig.5.