Reasoning-to-Defend: Safety-Aware Reasoning Can Defend Large Language Models from Jailbreaking

TL;DR

This paper introduces Reasoning-to-Defend, a safety-aware reasoning training paradigm for large language models that enhances their ability to defend against jailbreak attacks while maintaining performance.

Contribution

It proposes a novel safety-aware reasoning mechanism and Contrastive Pivot Optimization to improve LLMs' safety and robustness against jailbreaks.

Findings

R2D significantly mitigates jailbreak attacks.

Models maintain original performance levels.

Enhanced safety perception improves robustness.

Abstract

Large Reasoning Models (LRMs) have recently demonstrated impressive performances across diverse domains. However, how the safety of Large Language Models (LLMs) benefits from enhanced reasoning capabilities against jailbreak queries remains unexplored. To bridge this gap, in this paper, we propose Reasoning-to-Defend (R2D), a novel training paradigm that integrates a safety-aware reasoning mechanism into LLMs' generation process. This enables self-evaluation at each step of the reasoning process, forming safety pivot tokens as indicators of the safety status of responses. Furthermore, in order to improve the accuracy of predicting pivot tokens, we propose Contrastive Pivot Optimization (CPO), which enhances the model's perception of the safety status of given dialogues. LLMs dynamically adjust their response strategies during reasoning, significantly enhancing their safety capabilities defending jailbreak attacks. Extensive experiments demonstrate that R2D effectively mitigates various attacks and improves overall safety, while maintaining the original performances. This highlights the substantial potential of safety-aware reasoning in improving robustness of LRMs and LLMs against various jailbreaks.