REFLECTOR: Internalizing Step-wise Reflection against Indirect Jailbreak

TL;DR

Reflector is a two-stage framework that internalizes self-reflection in LLMs to effectively defend against complex jailbreak attacks and improve task performance.

Contribution

It introduces a novel two-stage training process combining supervised fine-tuning and reinforcement learning to internalize self-reflection in LLMs for safety and utility.

Findings

Achieves over 90% success rate against indirect jailbreak attacks.

Improves GSM8K performance by 5.85%.

Enhances robustness across diverse threat scenarios.

Abstract

While Large Language Models (LLMs) demonstrate remarkable capabilities, they remain susceptible to sophisticated, multi-step jailbreak attacks that circumvent conventional surface-level safety alignment by exploiting the internal generation process. To address these vulnerabilities, we propose Reflector, a principled two-stage framework that internalizes self-reflection within the generation trajectory. Reflector first leverages teacher-guided generation to produce high-quality reflection data for supervised fine-tuning (SFT), establishing structured reflection patterns. It subsequently uses Reinforcement Learning (RL) with outcome-driven and reward-validity supervision to instill robust, autonomous self-reflection capabilities. Empirical results show that Reflector achieves Defense Success Rates (DSR) exceeding 90% against complex indirect attacks while generalizing robustly across diverse threat scenarios. Notably, the framework enhances both task-specific and general utility, yielding a 5.85% gain on GSM8K alongside improved performance on knowledge-intensive benchmarks. By internalizing trajectory-level safety, Reflector overcomes the fundamental limitations of surface alignment without significant computational overhead, offering an efficient and scalable solution for the development of safe and capable LLMs.