From Threat to Tool: Leveraging Refusal-Aware Injection Attacks for Safety Alignment

TL;DR

This paper introduces RAAI, a simple, training-free framework that uses attack techniques to generate synthetic data, improving LLM safety alignment by making models more robust against harmful prompts.

Contribution

We propose RAAI, a novel, model-agnostic method that leverages refusal signals to inject harmful phrases, enabling scalable safety data generation without complex prompting or auxiliary models.

Findings

RAAI significantly increases harmful response rates in jailbreak tests.

Synthetic data from RAAI enhances LLM robustness against harmful prompts.

The method maintains model performance on standard benchmarks.

Abstract

Safely aligning large language models (LLMs) often demands extensive human-labeled preference data, a process that's both costly and time-consuming. While synthetic data offers a promising alternative, current methods frequently rely on complex iterative prompting or auxiliary models. To address this, we introduce Refusal-Aware Adaptive Injection (RAAI), a straightforward, training-free, and model-agnostic framework that repurposes LLM attack techniques. RAAI works by detecting internal refusal signals and adaptively injecting predefined phrases to elicit harmful, yet fluent, completions. Our experiments show RAAI effectively jailbreaks LLMs, increasing the harmful response rate from a baseline of 2.15% to up to 61.04% on average across four benchmarks. Crucially, fine-tuning LLMs with the synthetic data generated by RAAI improves model robustness against harmful prompts while preserving general capabilities on standard tasks like MMLU and ARC. This work highlights how LLM attack methodologies can be reframed as practical tools for scalable and controllable safety alignment.