Beyond SFT: Reinforcement Learning for Safer Large Reasoning Models with Better Reasoning Ability

TL;DR

This paper explores reinforcement learning as a method to improve safety and reasoning ability in large reasoning models, addressing limitations of supervised fine-tuning.

Contribution

It introduces RL-based safety training for LRMs, demonstrating improved safety and reasoning consistency over traditional supervised approaches.

Findings

RL achieves stronger safety improvements than SFT.

RL maintains reasoning ability better than SFT.

Analysis shows RL reduces unsafe reasoning trajectories.

Abstract

Large reasoning models (LRMs) extend large language models by generating explicit chain-of-thought (CoT) reasoning, significantly improving mathematical and logical problem solving. However, this explicit reasoning process also introduces new safety risks, as unsafe behaviors often emerge within intermediate reasoning trajectories, even when final answers appear harmless. Existing safety alignment approaches primarily rely on supervised fine-tuning (SFT) over safety-oriented long CoT datasets. While intuitive, we find that SFT produces inconsistent safety improvements, degrades reasoning ability, and generalizes poorly across model families. These limitations suggest that purely supervised approaches are insufficient for robust safety alignment in LRMs. To address this, we investigate reinforcement learning (RL) as a complementary optimization framework for LRM safety training. Unlike SFT, RL directly optimizes model policies with reward feedback, enabling more adaptive and stable alignment. Extensive experiments across multiple model families and benchmarks show that RL achieves stronger and more consistent safety gains while maintaining reasoning competence. Further analysis of reflection dynamics and token-level entropy reveals that RL suppresses unsafe exploratory reasoning while preserving reflective depth, leading to safer and more reliable reasoning processes.