Forewarned is Forearmed: Pre-Synthesizing Jailbreak-like Instructions to Enhance LLM Safety Guardrail to Potential Attacks

TL;DR

This paper introduces IMAGINE, a framework that synthesizes jailbreak-like instructions to improve large language models' safety guardrails against unseen malicious attacks by filling distributional gaps.

Contribution

It presents a novel embedding-based synthesis method that enhances safety training data, reducing jailbreak success rates without harming model utility.

Findings

Significant reduction in attack success rates on multiple LLMs.

Effective augmentation of safety data with synthesized jailbreak instructions.

Maintains model utility while improving safety defenses.

Abstract

Despite advances in improving large language model (LLM) to refuse to answer malicious instructions, widely used LLMs remain vulnerable to jailbreak attacks where attackers generate instructions with distributions differing from safety alignment corpora. New attacks expose LLMs' inability to recognize unseen malicious instructions, highlighting a critical distributional mismatch between training data and real-world attacks that forces developers into reactive patching cycles. To tackle this challenge, we propose IMAGINE, a synthesis framework that leverages embedding space distribution analysis to generate jailbreak-like instructions. This approach effectively fills the distributional gap between authentic jailbreak patterns and safety alignment corpora. IMAGINE follows an iterative optimization process that dynamically evolves text generation distributions across iterations, thereby augmenting the coverage of safety alignment data distributions through synthesized data examples. Based on the safety-aligned corpus enhanced through IMAGINE, our framework demonstrates significant decreases in attack success rate on Qwen2.5, Llama3.1, and Llama3.2 without compromising their utility.

Tables

Table 1. Table 1 : Jailbreak ASR ( A ttack S uccess R ate) on safety enhanced models. ORG: No DPO, original model; RW: Simply rewrite data; HH: Anthropic-HH  (Bai et al., 2022 ) ; HH-G: Anthropic-HH-Golden  (Cai et al., 2024 ) ; PKHF: Pku_SafeRLHF  (Ji et al., 2025 ) ; IMAGINE : Our proposed method.

DPO Dataset		Qwen2.5					Llama3.1					Llama3.2
		GCG	AutoDAN	GPTFUZZ	TAP	ReNeLLM	GCG	AutoDAN	GPTFUZZ	TAP	ReNeLLM	GCG	AutoDAN	GPTFUZZ	TAP	ReNeLLM
Act as Individual Alignment Dataset
	ORG	0.08	0.10	1.00	0.48	1.00	0.06	0.12	0.76	0.36	1.00	0.04	0.18	0.40	0.12	1.00
	RW	0.02	0.16	0.94	0.18	1.00	0.04	0.10	0.82	0.30	0.98	0.00	0.12	0.12	0.12	1.00
	HH	0.00	0.08	1.00	0.16	1.00	0.00	0.08	0.34	0.12	0.30	0.00	0.08	0.32	0.10	0.56
	HH-G	0.00	0.04	0.92	0.14	0.94	0.00	0.10	0.36	0.14	0.24	0.00	0.10	0.28	0.12	0.68
	PKHF	0.00	0.04	0.90	0.10	1.00	0.00	0.04	0.28	0.14	0.26	0.00	0.06	0.22	0.14	0.98
	IMAGINE	0.02	0.04	0.90	0.08	0.92	0.00	0.06	0.04	0.00	0.78	0.00	0.08	0.04	0.00	0.68
Act as Supplementary Data
HH	+None	0.00	0.08	1.00	0.16	1.00	0.00	0.08	0.34	0.12	0.30	0.00	0.08	0.32	0.10	0.56
	+RW	0.00	0.06	0.94	0.18	1.00	0.00	0.06	0.34	0.14	0.46	0.02	0.06	0.10	0.04	0.96
	+IMAGINE	0.00	0.04	0.10	0.08	0.82	0.00	0.02	0.06	0.02	0.14	0.00	0.02	0.02	0.06	0.72
HH-G	+None	0.00	0.04	0.92	0.14	0.94	0.00	0.10	0.36	0.14	0.24	0.00	0.10	0.28	0.12	0.68
	+RW	0.02	0.08	0.92	0.14	0.96	0.00	0.04	0.38	0.14	0.42	0.02	0.08	0.08	0.04	0.80
	+IMAGINE	0.00	0.02	0.10	0.06	0.86	0.00	0.00	0.02	0.02	0.12	0.00	0.04	0.04	0.00	0.68
PKHF	+None	0.00	0.04	0.90	0.10	1.00	0.00	0.04	0.28	0.14	0.26	0.00	0.06	0.22	0.14	0.98
	+RW	0.02	0.06	0.92	0.10	0.86	0.00	0.04	0.30	0.08	0.10	0.00	0.06	0.00	0.04	1.00
	+IMAGINE	0.00	0.00	0.00	0.00	0.84	0.00	0.02	0.00	0.02	0.10	0.00	0.02	0.00	0.04	0.34

Table 2. Table 2 : Evaluate on MMLU  (Hendrycks et al., 2021 ) subsets. We choose all subject areas to evaluate whether IMAGINE is malicious to the model’s helpfulness. PR: public relations; SY: sociology; MA: managements.

Model	DPO Dataset	MMLU  [Accuracy($△$)]
		PR	SY	MA
Qwen2.5	Original	0.70	0.84	0.80
	HH	0.70 (+0.00)	0.84 (+0.00)	0.83 (+0.03)
	PKHF	0.67 (-0.03)	0.83 (-0.01)	0.81 (+0.01)
	IMAGINE	0.70 (+0.00)	0.84 (+0.00)	0.83 (+0.03)
	HH+IMAGINE	0.67 (-0.03)	0.81 (-0.03)	0.78 (-0.02)
	PKHF+IMAGINE	0.67 (-0.03)	0.83 (-0.01)	0.79 (-0.01)
Llama3.1	Original	0.62	0.75	0.67
	HH	0.64 (+0.02)	0.77 (+0.02)	0.70 (+0.03)
	PKHF	0.62 (+0.00)	0.77 (+0.02)	0.75 (+0.08)
	IMAGINE	0.61 (-0.01)	0.73 (-0.02)	0.61 (-0.06)
	HH+IMAGINE	0.59 (-0.03)	0.80 (+0.05)	0.69 (+0.02)
	PKHF+IMAGINE	0.60 (-0.02)	0.77 (+0.02)	0.73 (+0.06)
Llama3.2	Original	0.30	0.26	0.16
	HH	0.23 (-0.07)	0.25 (-0.01)	0.17 (+0.01)
	PKHF	0.25 (-0.05)	0.26 (+0.00)	0.21 (+0.05)
	IMAGINE	0.21 (-0.09)	0.27 (+0.01)	0.20 (+0.04)
	HH+IMAGINE	0.26 (-0.04)	0.24 (-0.02)	0.21 (+0.05)
	PKHF+IMAGINE	0.22 (-0.08)	0.24 (-0.02)	0.18 (+0.02)

Table 3. Table 3 : ASR of classical jailbreak methods and IMAGINE generated data.

Method	Model
	Qwen2.5	Llama3.1	Llama3.2
GPTFUZZ	1.00	0.76	0.40
ReNeLLM	1.00	1.00	1.00
TAP	0.48	0.36	0.12
Ours	0.20	0.10	0.08

Table 4. Table 4 : ASR of attacks on models trained with data in different data-synthesis stages.

	Qwen2.5		Llama3.1		Llama3.2
	ReNeLLM	TAP	ReNeLLM	TAP	ReNeLLM	TAP
Original	1.00	0.48	1.00	0.36	1.00	0.12
Stage I Only	1.00	0.16	0.96	0.20	1.00	0.20
Stage II Only	1.00	0.20	0.98	0.30	1.00	0.12
Stage I& II	0.92	0.08	0.78	0.00	0.68	0.00

Table 5. Table 5 : Ablation results on loss functions in stage I and stage II.

	Qwen2.5		Llama3.1		Llama3.2
	ReNeLLM	TAP	ReNeLLM	TAP	ReNeLLM	TAP
Original	1.00	0.48	1.00	0.36	1.00	0.12
In Stage I
Only $L_{\text{sft}}$	0.92	0.20	0.92	0.04	0.74	0.04
Only $L_{\text{sem}}$	0.92	0.40	0.96	0.32	0.98	0.08
$L_{\text{sft}},L_{\text{sem}}$	0.92	0.08	0.78	0.00	0.68	0.00
In Stage II
Only $L_{\text{con}}$	0.92	0.20	0.82	0.04	0.74	0.04
Only $L_{\text{top}}$	1.00	0.18	0.98	0.18	1.00	0.2
Only $L_{\text{dis}}$	0.98	0.16	0.94	0.20	1.00	0.22
$L_{\text{con}},L_{\text{top}}$	0.92	0.40	0.96	0.32	0.98	0.08
$L_{\text{con}},L_{\text{dis}}$	0.94	0.16	0.94	0.18	0.94	0.04
$L_{\text{top}}+L_{\text{dis}}$	0.96	0.20	0.96	0.24	1.00	0.20
$L_{\text{con}},L_{\text{dis}},L_{\text{top}}$	0.92	0.08	0.78	0.00	0.68	0.00

Table 6. Table 6 : Performance comparison of ReNeLLM and TAP across different settings.

	llama3.1		llama3.2
	ReNeLLM	TAP	ReNeLLM	TAP
$a=0.2\left(b=0.8\right)$	1.00	0.42	1.00	0.16
$a=0.4\left(b=0.6\right)$	1.00	0.26	0.86	0.06
$a=0.6\left(b=0.4\right)$	0.88	0.12	0.72	0.08
$a=0.8\left(b=0.2\right)$	0.78	0.00	0.68	0.00
$a=1.0\left(b=0.0\right)$	0.82	0.04	0.72	0.00

Table 7. Table 7 : Ablation results of ASR. The value pair in each cell confroms to the form of ASR ReNeLLM | ASR TAP \text{ASR}_{\text{ReNeLLM}}|\text{ASR}_{\text{TAP}} .

	$\gamma =\text{0.2}$	$\gamma =\text{0.4}$	$\gamma =\text{0.6}$	$\gamma =\text{0.8}$
Llama3.1-8B
$\delta =\text{0.2}$	0.96 | 0.24	0.86 | 0.16	0.80 | 0.10	0.82 | 0.02
$\delta =\text{0.4}$	0.90 | 0.12	0.76 | 0.08	0.86 | 0.06	—
$\delta =\text{0.6}$	0.78 | 0.00	0.82 | 0.10	—	—
$\delta =\text{0.8}$	0.82 | 0.04	—	—	—
Llama3.2-3B
$\delta =\text{0.2}$	0.82 | 0.08	0.88 | 0.10	0.72 | 0.04	0.76 | 0.08
$\delta =\text{0.4}$	0.78 | 0.04	0.70 | 0.08	0.76 | 0.08	—
$\delta =\text{0.6}$	0.68 | 0.00	0.80 | 0.02	—	—
$\delta =\text{0.8}$	0.76 | 0.04	—	—	—

Table 8. Table 8: Mean score on full MMLU subsets.

Dataset	Model
	Qwen2.5	Llama3.1	Llama3.2
ORG	0.677	0.530	0.258
HH	0.682	0.539	0.256
PKHF	0.655	0.557	0.256
HH+Ours	0.635	0.534	0.266
PKHF+Ours	0.658	0.543	0.254

Equations

$$\theta=\mathop{\arg\min}\limits_{\theta}\bigl{[}\alpha\cdot L_{\text{sft}}(q^{\prime},\hat{q})-\beta\cdot L_{\text{sem}}(q^{\prime},\hat{q})\bigr{]},$$

$$L_{\text{sft}}(q^{\prime },\hat{q})$$

$$L_{\text{sem}}(q^{\prime },\hat{q})$$

$$L_{\text{aug}}=$$

$$+$$

$$L_{\text{con}}^i=\sum _j{L}_{\text{con}}^{i,j},$$

$$L_{\text{con}}^{i,j}(Z_{{\tilde{x}}_j},Z_{{\tilde{x}}_t})$$

$$-\frac{1}{N}\sum _{k=1}^N||z_{{\tilde{x}}_{t,j}}-z_{{\tilde{x}}_{i,k}}|{|}^2,$$

$$L_{\text{topic}}$$

$$L_{\text{dis}}=(||$$

Full text

= Date: August 27, 2025

= Venue: EMNLP 2025 Findings

1 Introduction

Recent research has significantly boosted the generative capabilities of large language models (LLMs), leading to the emergence of models such as GPT-4, DeepSeek-V3/R1, and Qwen2.5. To make LLMs capable of clearly refusing to generate answers containing unsafe contents (Li et al., 2025b) and misinformation (Liu et al., 2024a, Hu et al., 2025) induced by socially malicious questions, developers have utilized various training methods such as Reinforcement Learning from Human Feedback (RLHF) (Dai et al., 2024, Bai et al., 2022), Supervised Fine-Tuning (SFT) (Bianchi et al., 2024, Qi et al., 2024), and Direct Preference Optimization (DPO) (Rafailov et al., 2024, Amini et al., 2024) for safety alignment. However, researchers and public forums continuously propose effective ways to bypass safety guards Mehrotra et al. (2023), Ding et al. (2024), Liu et al. (2024d), requiring model developers to constantly address these carefully designed attacks in time.

Addressing new jailbreaking vulnerability through “post-hoc” fixes shows evident lag, as it can only collect jailbreaking data for analysis after a new attack method is exposed, and then fine-tune the model to refuse to answer. This “one step slower” delay naturally leads to the model producing malicious output before specific fixes are implemented, causing significant malicious social influence. According to Zhou et al. (2024a), current jailbreak attacks consist of four processes: select, mutate, constrain, and evaluate. Different jailbreaking methods generate diverse attack instructions due to the differences in these parts, making it costly for model developers to fix (Wei et al., 2024, Deng et al., 2025b) and difficult for the static safety alignment corpus to cover jailbreaking data distributions. Such a situation raises the question: Can we simulate jailbreak attacks before they occur?

Exploring this possibility involves both opportunities and challenges. Advances in adversarial training suggest it may be feasible to generate synthetic attack data from known patterns Yang et al. (2023), Liu et al. (2024c), Xue et al. (2024), helping anticipate emerging methods. Although real-world attacker creativity and risks like overfitting could limit its effectiveness, proactively generating such data could indeed reduce costs and delays of reactive fixes, mitigating the “one step slower” situation. In this paper, we propose a generation technique IMAGINE (Iterative Malicious data Generation In Embedding Space) to synthesize jailbreak-like data. By synthesizing jailbreak-like texts, we enhance the model’s ability to recognize real jailbreaking texts and clearly refuse to answer such malicious questions without doing harm to their original helpfulness. Overall, our contributions are as follows:

• •

Idea: Different from traditional safety alignment corpora construction method, we introduce an embedding-driven adversarial text generation framework to synthesize jailbreak-like instructions.

• •

Method: IMAGINE bridges the distributional gap between safety alignment corpora and malicious instructions by dynamically generating adversarial examples in the model’s latent space, thereby enriching the diversity of safety-aligned training data.

• •

Effectiveness: The proposed method enables implicit expansion of the model’s rejection boundaries to encompass malicious intents with maximum ASR decrease of 90% on GPTFUZZ while preserving functional integrity, achieving enhanced safety without compromising model’s basic capabilities.

2 Related Works

Jailbreak Attack LLM jailbreak focuses on inducing the model to generate malicious responses against the usage of model policy and society by carefully designing adversarial instructions. According to different methodologies, jailbreak methods can be divided into three categories: inner-state-based,finetuning-based, and prompt-based. Inner-state-based methods optimize attack instructions by inserting probes to get model’s running hidden states and find vulnerabilities. Zou et al. (2023) propose an effective gradient-based jailbreak attack method Greedy Coordinate Gradient (GCG) to manipulate model inputs based on pseudo-gradient to elicit a compliant response to malicious instructions. Liu et al. (2024d) generate the attack tokens as suffixes using the Single Token Optimization (STO) algorithm that considers both jailbreak and readability objectives. Finetuning-based methods train the target LLM with malicious QA examples to elicit malicious behaviors. Qi et al. (2024) demonstrate that when finetuning LLMs using merely a small number of malicious examples, it can severely undermine their safety alignment. Prompt-based methods include completing harmless prompt templates with malicious questions, such as completing an unfinished code with malicious instructions as variable name. Kang et al. (2024) segregate sensitive lexicons into variables within templates to hide malicious intent. These jailbreak methods rely on scenarios where malicious intent can be concealed, highlighting that defending against straightforward malicious queries is far from addressing real-world attacks.

Jailbreak Defense As LLM jailbreak attack techniques progress and safe issues become intense, a variety of defense strategies have been put forward to safeguard language models against possible attacks. Those strategies can be roughly categorized into prompt-level defense and model-level defense. Prompt-level defense aims to filter or preprocess suspicious instructions and render them less malicious. Jain et al. (2023) propose a threshold-based detection method that computes the perplexity of both the text segments and the entire instruction in the context window, and then declares the harmfulness of the input texts. Robey et al. (2023) apply character-level perturbation to the copies of given instructions and select a final instruction that consistently defends the jailbreak attack. Model-level defense aims to help the safety guardrail generalize better to address a wide variety of attacks. RLHF (Dai et al., 2024, Bai et al., 2022, Ji et al., 2025, Qi et al., 2025, Deng et al., 2025a) is a widely-used model training procedure applied to a well-pretrained language model to further align model behavior with human preferences and instructions. Recent works propose DPO (Rafailov et al., 2024, Amini et al., 2024), a more stable and lightweight method, as a substitute of RLHF. Those training-based methods require lots of data in practice, thus many efforts have been put into constructing a safety alignment dataset emphasizing safety and trustworthiness. Bianchi et al. (2024) discuss how the mixture of safety data and target instruction affects safety. However, their safety training corpus construction processes ignore the differences in real attack contents. Our proposed IMAGINE is tailored to this challenge and achieves better align performance than traditional alignment corpora in enhancing the model safety guardrail.

3 Method

We tackle the problem that current jailbreaking texts have a different distribution from the safety alignment corpus by simulating the potential jailbreaking texts through a set of data augmentation models. At a high level, for jailbreaking data simulation, two key factors should be considered: 1) harmless scenario generation, *i. e. *, how to train the model to generate different safety scenarios as it determines the content diversity of synthesized data and the similarity to real jailbreak instructions; 2) diversified data distribution. Given an intent-concealed base model, how to synthesize diversified data to efficiently cover as much latent space as possible.

Considering the two key factors, we design a two-stage optimization process in IMAGINE. In the first stage, a generative model $G$ is trained to become intent-concealing base model $M_{0}$. $M_{0}$ conceals malicious intents of instructions as by inserting raw instructions into various scenarios. In the second stage, based on $M_{0}$, we iteratively train $k$ variants of $M_{0}$ with meticulously designed constraints. In following subsections, we describe the two stages in detail.

3.1 Stage I: Train Intent-Concealing Model

Current LLMs have been strictly aligned with ethical values to enhance their sensitivity to malicious instructions, thereby enabling them to refuse direct malicious user instructions. However, existing methods such as PAIR (Chao et al., 2023), and ReNeLLM (Ding et al., 2024) show that using a set of manually crafted templates to wrap the original malicious instructions can effectively induce LLMs to give responses.

To simulate the handcrafted scenarios, we collect data from APRT (Jiang et al., 2025), which is composed of pairs of direct instruction and intent-concealed instruction and use DeepSeek-V3 (Liu et al., 2024b) model to rewrite questions, enlarging the original dataset. In the rewriting process, we use the designed prompt shown in Appendix B to ensure that the intent of direct instruction is kept in the model’s output and the output scenario is different from paired the intent-concealed instruction. Upon finishing the train data collection, we adopt SFT to train an intent-concealing model.

Formally, given simple instruction $q$ and its paired intent-concealed instruction $\hat{q}$, the target of model $G(\theta,q)$ is to find $\theta$ satisfying:

[TABLE]

where $\alpha$ and $\beta=1-\alpha$ are hyper-parameters for balancing two losses. $L_{\text{sft}}(q^{\prime},\hat{q})$ and $L_{\text{sem}}(q^{\prime},\hat{q})$ are defined as follows:

[TABLE]

[TABLE]

$L_{\text{sft}}$ ensures that the model can learn to conceal intentions while $L_{\text{sem}}$ computes the latent space distance between sentence semantic embedding $z^{{}^{\prime}}\ \text{and}\ \hat{z}\in\mathbb{R}^{d}$, ensuring diverse scenario could be generated rather than simply outputting scenarios in the training data.

In our implementation, We use Qwen2.5 (Yang et al., 2024) model instead of BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) as our semantic extractor to extract semantic embedding $z$ for given instruction $q$ due to the long context length and complicated semantics of intent-concealed texts.

3.2 Stage II: Iterative Latent Space Expansion

As mentioned above, jailbreaking data and alignment data follow different distributions. Consequently, we utilize an iterative training process to expand the hidden space of the synthesized simulation data, forming $k$ distributed groups of jailbreak-like data where $k$ is a hyper-parameter denoting the number of iterations and distributions. Figure 2c shows the framework of ISE. ISE initially contains the intent-concealing base model $M_{0}$, and $k$ variants of $M_{0}$ as augmentation models are progressively trained. The base model $M_{0}$ gives its variants the ability to finish the intent-concealing task, while augmentation models $\{M_{i}\}_{i=1}^{k}$ generate different distributed jailbreak-like texts to expand the latent space that synthesized data possess.

Training augmentation models  Let $S=\{s_{1},s_{2},\cdots,s_{n}\}$ be the dataset containing $n$ safe instructions, $H=\{h_{1},h_{2},\cdots,h_{n}\}$ be the dataset consisting of $n$ straightforward malicious instructions, and $\tilde{X_{i}}=\{\tilde{x}_{i,1},\tilde{x}_{i,2},\cdots,\tilde{x}_{i,n}\}$ where $i\in\{1,2,...,k\}$ be the augmented texts generated in iteration $i$. In iteration $t$, the overall loss function for present augmentation model is as follows:

[TABLE]

where $L_{\text{topic}}$ is sentence topic loss between augmented and input instructions in embedding space, $L_{\text{con}}$ is contrastive losses between data generated in iteration $i$ and data in iteration [math] to $t-1$, and $L_{\text{dis}}$ is distance loss among safe instructions and malicious instructions. $Z_{x}$ denotes a set of corresponding semantic embeddings of dataset $X$. $\delta$, $\phi$ and $\gamma=1-\delta-\phi$ are hyper-parameters.

To expand the latent space occupied by the generated data, an intuitive way is to increase the number of augmentation models. However, naively increasing the number of augmentation models by training them independently could result in the generated data from different iterations having similar latent space distribution (Yang et al., 2023, Nan et al., 2024). Thus, we propose to increase the diversity iteratively by contrastive loss (Khosla et al., 2020), constraining the discrepancy of old and new augmented data at each iteration, narrowing the data embedding in the current iteration, and keeping the embedding away from those previous iterations. Also, simply increasing the embedding distance without other direction would produce meaningless content, topic deviation texts, and trivial output similar to a simple malicious instruction. Consequently, we apply the topic loss and distance loss to limit the outputs to be malicious, meaningful, and non-trivial:

[TABLE]

where $d$ is a distance margin to avoid generating text identical to the given malicious instruction, and $m$ denotes the number of samples. By optimizing $L_{\text{aug}}$, the original data distribution is expanded in $k$ iterations, and the generated $k$ sets of data are distributed differently in latent space.

4 Experiments

In this section, we conduct experiments to answer the following questions:

RQ1: Can IMAGINE itself be utilized to further strengthen the safety guardrails of aligned models?

RQ2: Can IMAGINE-generated data serve as a supplement, enhancing existing safety alignment corpora and improving alignment performance?

RQ3: To achieve optimal performance, how many iterations of IMAGINE are needed to effectively expand the space?

4.1 Experimental Details

Datasets We choose a set of well-established benchmarks for evaluating IMAGINE effectiveness. These datasets provide diverse and representative tasks to ensure evaluations are fair and comprehensive. See Appendix A for details.

Models In IMAGINE, we use Qwen2.5-7B-Instruct as $G$ and train it to be the intent-concealing base model $M_{0}$. For extracting sentence embedding, we choose Qwen2.5-7B-Base and collect its hidden embedding at the last token from layer 10 (Zhou et al., 2024b) to serve as the text embedding for safe/harmful instructions and newly generated samples. To evaluate the effectiveness of IMAGINE-augmented data, we select Llama-3.1-8B-Instruct, Llama-3.2-1B-Instruct, and Qwen2.5-7B-Instruct to be trained with alignment data.

Iterative Space Expansion Training Setting The training progress is shown in Algorithm 1. By expansion, we collect $k$ groups of augmented jailbreak-like data pairs of simple instruction and intent-concealed instruction.

Alignment Training Setting We use vanilla DPO Rafailov et al. (2024) as the basic safety alignment method. In DPO, we randomly sample 3,000 entries from Anthropic-HH (Bai et al., 2022), Anthropic-HH-Golden (Cai et al., 2024), Pku_SafeRLHF (Ji et al., 2025), and IMAGINE-generated data, respectively, and also generate 3,000 entries through simple rewriting (see Appendix B) additionally for comparison.

Hyper-parameters For training intent concealing model in stage I, $\alpha,\beta$ are set to 0.8, 0.2, respectively, with 600 pairs of simple and intent-concealed instructions. For ISE training in stage II, $\delta$, $\phi$, $\gamma$ are set to 0.6, 0.2, 0.2 respectively, and the number of classes $k$ is set to 6 according to Section 4.2.2 and $8\times(t-1)$ samples at iteration $t$ are selected to calculate $L_{\text{con}}$. We train each model by DPO for two epochs on each alignment dataset and choose the best model to jailbreak and evaluate.

Jailbreak Methods We choose GCG (Zou et al., 2023), AutoDAN (Liu et al., 2024d), GPTFUZZER (Yu et al., 2024), ReNeLLM (Ding et al., 2024), and TAP (Mehrotra et al., 2023) as our jailbreak attack methods, and we follow their default settings in our experiments.

Evaluation Metric ASR (Attack Success Rate) is widely used to evaluate model vulnerabilities when encountering jailbreak attacks. Previous works mainly adopted two schemes to calculate ASR: keywords matching and auxiliary model judging. The former uses a manually selected set of refusal words/phrases like “sorry” and “as a responsible AI” to verify whether a response is harmful or not while the latter uses an advanced LLM to verify. Considering the high false positive rate of keyword matching, leading to an extremely high ASR, we use the GPT-4o-0718 and set six levels to score the harmfulness of each answer. An answer is considered to be harmful if and only if it follows the instruction and provides relevant harmful details. More Details are shown in Appendix D.

4.2 Experimental Results

4.2.1 Effectiveness of IMAGINE

For RQ1, we aim to see whether IMAGINE itself could further enhance the model safety guardrail. Table 1 shows that IMAGINE outperforms in all models and jailbreak methods with a max decrease 0.72 in ASR compared to other safety alignment corpora. The decline indicates that IMAGINE data effectively enhances the model safety guardrail by importing diverse data distribution in DPO training rather than simply enlarging the data scale.

For RQ2, we train those models on vanilla and IMAGINE data added safety alignment dataset, respectively. By adding IMAGINE data into Anthropic-HH, Anthropic-HH-Golden, and Pku_SafeRLHF, models perform better than those trained with vanilla datasets, suggesting that the diversity in vanilla datasets is not enough to protect LLMs from real jailbreak attacks and IMAGINE could act as a supplement by introducing latent-space-close jailbreak-like data.

Latent Space Visualization To have a better understanding of the effects IMAGINE takes on, we visualize embeddings of the data on the model’s latent space. In Figure 3, not only does IMAGINE cover most space of the safety alignment datasets, but also IMAGINE is closer to real jailbreak attacks in latent space than safety alignment datasets. This give reasonable explanations for the dual function IMAGINE has, as it is not a similar copy of the previous safety alignment corpus, but a “filling of the blank space”, resulting in good performance on enhancing model safety guardrail both separately (RQ1) and jointly  (RQ2).

4.2.2 Best Expanding Iterations

As the iteration progresses, the time consumed for each iteration gradually becomes longer due to the cost of computing $L_{\text{con}}$. Also, considering the problem that if $k$ is set to a large number, newly generated examples will face degradation of the $L_{\text{con}}$ (Khosla et al., 2020), since samples synthesized previously will dominate the $L_{\text{con}}$. Thus, for RQ3, we aim to discover the least iterations needed to generate enough supplementary data, balancing the computation cost and performance.

In practice, we set the number of iterations $k$ from 0 to 10 at an interval of 2 to test the effectiveness of enhancing model safety guardrails respectively. As shown in Figure 4, as $k$ increases, the ASR keeps decreasing among the two models, indicating larger $k$ would be better for safety alignment, but when $k\geq 6$, the trends slow down with nearly $5\%$ decrease of ASR. Therefore, we set $k=6$ in our experiments without specification.

4.2.3 Side Effects Evaluation

Influence to Model Capability From Yang et al. (2025) and Xu et al. (2025), addressing safety issues sometimes leads model to give uninformative answers (e.g., “I don’t know”) to most questions. Thus, we use MMLU (Hendrycks et al., 2021) to evaluate the model’s ability to solve normal daily questions. We use all topics in the dataset for meticulousness, and part of the results are shown in Table 2. The maximum decrease of the model is less than $0.1$, suggesting that aligning with IMAGINE data won’t do damage to the model’s capabilities to daily questions. Full results of MMLU Hendrycks et al. (2021) are shown in Appendix F.

Harmfulness to Model Security Considering that synthesizing jailbreak-like data can be a “double-edged sword”, here, we evaluate the ASR of our generated data in comparison to real jailbreak methods to see whether our generated data can be malicious to LLMs. The results are shown in Table 3. Compared to those jailbreak methods, IMAGINE achieves the least ASR, indicating IMAGINE is not an effective attack tool.

4.3 Ablation Study

There are three ablations conducted in our experiment. The first ablation is to identify the effectiveness of stage I and stage II. The second ablation is to figure out the influence of the loss functions on the final result. The third ablation study conducts experiments on the values of hyper-parameters.

4.3.1 Individual Contribution of Each Stage

Table 4 shows the necessity of each stage. It shows that separately conducting DPO on data generated solely on stage I and stage II can slightly decrease the ASR, though, the combination of these two stages in IMAGINE achieves the best performance. Also, the result indicates that intent-concealing plays a more important role than different expressions in jailbreak attacks.

4.3.2 Loss Function Analysis

Table 5 shows the influences each function have on the final result in each stage. As we designed, $L_{\text{sem}}$ in stage I is to enrich the scenarios and prevent overfitting to scenarios in dataset; $L_{\text{top}}$ and $L_{\text{dis}}$ in stage II are aimed at controlling the distance among newly generated jailbreak-like data, safe instructions and straightforward harmful instructions.

4.3.3 Hyper-parameter Analysis

IMAGINE contains five primary hyper-parameters, i.e., $\alpha,\beta$ in Stage I and $\delta,\phi,\gamma$ in Stage II. Under the designed constraints that $\alpha+\beta=1\text{ and }\delta+\phi+\gamma=1$, for those 3 free hyper-parameters ($\beta$ in stage I; $\delta$ and $\gamma$ in stage II), we conduct experiments with their value ranging in 0.2, 0.4, 0.6, 0.8, respectively to evaluate its impact to the final alignment results. The results are shown in Tables 6 and 7.

5 Conclusion

This paper tackles a problem in LLM’s safety alignment when collecting data to enhance the model safety guardrail, the distribution gap problem that the safety alignment corpus follows a completely different latent space distribution from the actual jailbreak attacks. We propose an effective iterative latent space expansion technique to simulate real jailbreak text distribution, making the model “forwarned” to potential attacks by expanding the data latent space. The experimental results indicate that our proposed method could act separately in enhancing model security, also as a supplementary to existing safety alignment corpus.

6 Discussion

Our proposed IMAGINE presents a new perspective on synthesizing safety alignment data for LLMs. By controlling the latent space distribution of synthesized data, we can generate diversified data that is more representative and closer to real-world attacks. Future work is expected to increase the diversity of generated attack texts using advanced generation techniques and multi-modal data. Moreover, dynamic safety alignment mechanisms, such as online or reinforcement learning, could help models adapt to evolving attack strategies more rapidly.

Limitations

We identify the following limitations of this work. First, the IMAGINE still relies on a set of paired simple and intent-concealed instructions, which may not fully capture the evolving nature of real-world attacks. Second, while our approach effectively expands the latent space of safety alignment data, the generated texts may still lack the full complexity and diversity found in real malicious instructions. Third, the computational cost associated with iterative latent space expansion remains a challenge, especially for large-scale deployment. Last, limited by computation resources, we did not validate our method on larger open-source models such as Qwen2.5-72B and DeepSeek-V3.

Ethical Consideration

Application developers are constantly building new tools powered by LLMs, which must adhere to ethics and fairness standards. They require approaches that enable them to integrate responsible AI practices and conduct adversarial testing early in the development process. While this research aims to improve AI safety, it also carries potential safety risks in:

• •

Malicious instruction generation: This is a double-edged sword effect that the IMAGINE-generated data could be used for safety alignment by model developers but for attacks by malicious attackers.

• •

Exposing vulnerabilities for current LLMs: As IMAGINE could analyze and expand the latent space distribution of the target model, it could be misused by malicious attackers generating instructions that get across the safety guardrail.

• •

Controlled Access: The full synthesized dataset of harmful instructions is never publicly released. Only sanitized subsets (with toxic content removed) or safety-oriented metadata (e.g., toxicity labels) are shared for research reproducibility.

• •

Security through Obscurity: We intentionally limit algorithmic details (e.g., specific prompt templates, optimization thresholds) that could lower the barrier for reverse-engineering attacks.

Our work is firmly grounded in ethical principles. It is specifically designed to have a beneficial impact on the realms of AI safety and research. We are convinced that the advantages derived from this research in promoting AI safety far exceed any potential risks. As the field of AI continues to progress and develop, maintaining unwavering vigilance and implementing responsible disclosure practices remain of utmost importance.

Appendix A Details of Datasets

A.1 Datasets For Training

• •

APRT (Jiang et al., 2025): The dataset consists of 300 manually paired malicious instructions and corresponding intent-concealed attack instructions. For a better trained intent-concealing model, empirically, we expand the dataset by using Deepseek-V3 model111https://github.com/deepseek-ai/DeepSeek-V3. After expanding, we manually check the pairs in the augmented dataset to ensure that the simple instruction is malicious and that intent-concealed instruction keeps the intent with a different expressions. In total, we use 600 paired data to train our intent concealing model.

• •

HarmBench (Mazeika et al., 2024): The dataset contains 510 unique malicious behaviors, split into 400 textual behaviors and 110 multimodal behaviors. The behaviors are designed to violate laws or norms, such that most reasonable people would not want a publicly available LLM to exhibit them. In our dataset, we use its textual behaviors subset as simple malicious instruction $H$.

• •

HelpSteer (Wang et al., 2024): The HelpSteer dataset comprises 10,459 single-turn instructions designed to enhance the robustness and diversity of language model training. These instructions were sourced through a combination of human-authored and synthetically generated methods: approximately half were created by Scale AI, an external data annotation provider, while the remaining half were algorithmically generated using templated approaches to ensure broad coverage of instruction styles and intents. In our experiments, we sample 400 safe instructions as our safe instruction dataset $S$.

A.2 Datasets For Evaluating

• •

Anthropic-HH (Bai et al., 2022): The Anthropic-HH (Harmless and Helpful) dataset is a benchmark designed to evaluate and improve the alignment of large language models with human values. We randomly sample its subset harmless-base, which contains 42,547 labeled preference data points, as one source of our DPO alignment corpora.

• •

Anthropic-HH-Golden (Cai et al., 2024): This dataset is constructed to test the ULMA technique. In the dataset, a new preference dataset with 44,849 examples is selected as an extension of the original Anthropic-HH dataset by providing more harmless answers as preferred chosen answers with GPT4.

• •

Pku_SafeRLHF (Ji et al., 2025): The dataset consists of 83.4K preference entries in QA format, which is annotated across two dimensions: harmlessness and helpfulness.

• •

MMLU (Hendrycks et al., 2021): The Massive Multitask Language Understanding (MMLU) dataset is a comprehensive benchmark designed to evaluate the knowledge and reasoning abilities of language models across 57 tasks spanning domains such as humanities, STEM and other fields. By evaluating on MMLU, we investigate whether the model’s usability is compromised before and after training.

• •

Advbench (Zou et al., 2023): AdvBench is a set of 500 malicious behaviors formulated as instructions and each data sample in Advbench is used as initial jailbreak instruction.

Appendix B Rewriting data from Malicious Instruction

In DPO, for comparison, we use Deepseek-V3 to simply rewrite malicious corpus $H$ to exclude the influence of enlarging the scale of safety alignment dataset. The conversation template for rewriting is shown in Figure 5.

Appendix C Intent Concealing Template

To expand the corpus of instruction pairs, we employ the Deepseek-V3 model to generate more pairs of simple instructions and intent-concealed instructions, drawing on the data provided in  Jiang et al. (2025). The prompt template in shown in Figure 6.

Appendix D Jailbreak Evaluation Template

Following the LLM-as-a-judge paradigm Li et al. (2025a), Hu et al. (2024), we adopt GPT-4o-mini-0718 as the judge model when running jailbreak methods. In order to quantify the harmfulness of the answers, we use six levels 1 (harmless), 3, 5, 6, 8, and 10 (extremely malicious) to rate the answers. Answers rated greater than 6 will be considered malicious. The evaluation conversation template is shown in Figure 10.

Appendix E Rationale for Model Selection

We justify our selection of Qwen2.5, Llama3.1, and Llama3.2 through the following considerations:

• •

Popularity and Community Adoption: The Llama series (v3.1/3.2) from Meta AI ranks among the most cited open-source LLMs, with over 500k cumulative downloads on HuggingFace Hub. Qwen2.5, developed by Alibaba Cloud, dominates Chinese-language model rankings while gaining significant international traction (Top-5 on OpenLLM Leaderboard).

• •

Training Cost Efficiency: Compared to similarly-sized counterparts, Llama3.2 achieves 1.8$\times$ faster training convergence using improved architectural choices. Qwen2.5’s hybrid precision training reduces GPU memory requirements by 35%, making it accessible for medium-scale deployments.

Disclaimer: While numerous capable models exist, this selection optimizes technical comparability, resource efficiency, and reproducibility.

Appendix F Full MMLU Evaluation

The full evaluations results are shown in Figure 11 and Table 8. We plot the differences compared to original model. As shown in Figure 11, our IMAGINE does not hurt model capability the model trained on IMAGINE-generated data has similar trends as those trained on other datasets.

Appendix G Cases of IMAGINE

Figure 7 shows an example of IMAGINE generated data. Intuitively, we can see that the model is concealing the intent of original instruction with more and more complex scenarios. More cases are shown in Figures 7,  8, and 9.