Self-ReSET: Learning to Self-Recover from Unsafe Reasoning Trajectories

TL;DR

Self-ReSET introduces a reinforcement learning framework that improves large reasoning models' ability to recover from unsafe trajectories, especially under adversarial attacks, by reusing their own failure states for training.

Contribution

It presents a novel reinforcement learning approach enabling models to self-recover from unsafe reasoning errors, addressing limitations of static training data.

Findings

Significantly improves robustness against adversarial attacks.

Enhances model recovery from unsafe intermediate states.

Maintains general utility while increasing safety.

Abstract

Large Reasoning Models possess remarkable capabilities for self-correction in general domain; however, they frequently struggle to recover from unsafe reasoning trajectories under adversarial attacks. Existing alignment methods attempt to mitigate this vulnerability by fine-tuning the model on expert data including reflection traces or adversarial prefixes. Crucially, these approaches are often hindered by static training data which inevitably deviate from model's dynamic, on-policy reasoning traces, resulting in model hardly covering its vast generation space and learning to recover from its own failures. To bridge this gap, we propose Self-ReSET, a pure reinforcement learning framework designed to equip LRMs with the intrinsic capacity to recover from their own safety error trajectories, which are subsequently reused as an initial state for reinforcement learning. Extensive experiments across various LRMs and benchmarks demonstrate that Self-ReSET significantly enhances robustness against adversarial attacks especially out-of-distribution (OOD) jailbreak prompts while maintaining general utility, along with efficient data utilization. Further analysis reveals that our method effectively fosters self-recovery patterns, enabling models to better identify and recover from unsafe intermediate error states back to benign paths. Our codes and data are available at https://github.com/Ing1024/Self-ReSET.