Self-Guided Defense: Adaptive Safety Alignment for Reasoning Models via Synthesized Guidelines

TL;DR

This paper presents SGASA, a novel framework that enhances reasoning models' safety by internalizing synthesized safety guidelines, enabling adaptive defense against adversarial prompts while maintaining performance on benign tasks.

Contribution

The paper introduces SGASA, a new adaptive safety alignment method that synthesizes safety guidelines and fine-tunes models to improve robustness against adversarial prompts.

Findings

SGASA significantly reduces harmful outputs in adversarial scenarios.

The framework maintains high performance on benign tasks.

Experiments validate SGASA's scalability and effectiveness.

Abstract

Reasoning models have demonstrated remarkable capabilities in complex reasoning tasks. However, ensuring their safety against adversarial jailbreak prompts remains a critical challenge. Due to the covert and deceptive nature of such prompts, they can often evade built-in safety mechanisms and lead to the generation of harmful content. This underscores the need for an adaptive safety alignment approach that enables models to autonomously reinforce their defenses in response to adversarial inputs. This paper introduces the Synthesized Guideline-based Adaptive Safety Alignment (SGASA) framework, which internalizes model-generated safety guidelines to strengthen models' ability to enhance robustness against harmful adversarial prompts while minimizing unnecessary refusals of benign requests. SGASA consists of two key stages: Data Pre-synthesis, which generates safety guidelines and augmented prompts; and Alignment Fine-tuning, which leverages Supervised Fine-tuning (SFT) and Direct Preference Optimization (DPO) to embed these guidelines into the model. Extensive experiments across multiple datasets demonstrate that SGASA significantly improves model safety, validating its adaptive and scalable effectiveness.