Mitigating Safety Tax via Distribution-Grounded Refinement in Large Reasoning Models

TL;DR

This paper introduces DGR, a method to refine safety datasets for large reasoning models, effectively reducing safety-related performance degradation by aligning data distributions, and reveals that safety activation may rely on latent knowledge activation with minimal samples.

Contribution

The paper proposes DGR, a novel dataset construction method that aligns safety reasoning data with the target model's distribution, improving safety and reasoning accuracy.

Findings

DGR reduces safety tax while maintaining safety performance.

Bridging distribution shift preserves reasoning capabilities.

Minimal samples activate safety refusal behaviors.

Abstract

Safety alignment incurs safety tax that perturbs a large reasoning model's (LRM) general reasoning ability. Existing datasets used for safety alignment for an LRM are usually constructed by distilling safety reasoning traces and answers from an external LRM or human labeler. However, such reasoning traces and answers exhibit a distributional gap with the target LRM that needs alignment, and we conjecture such distributional gap is the culprit leading to significant degradation of reasoning ability of the target LRM. Driven by this hypothesis, we propose a safety alignment dataset construction method, dubbed DGR. DGR transforms and refines an existing out-of-distributional safety reasoning dataset to be aligned with the target's LLM inner distribution. Experimental results demonstrate that i) DGR effectively mitigates the safety tax while maintaining safety performance across all baselines, i.e., achieving \textbf{+30.2\%} on DirectRefusal and \textbf{+21.2\%} on R1-ACT improvement in average reasoning accuracy compared to Vanilla SFT; ii) the degree of reasoning degradation correlates with the extent of distribution shift, suggesting that bridging this gap is central to preserving capabilities. Furthermore, we find that safety alignment in LRMs may primarily function as a mechanism to activate latent knowledge, as a mere \textbf{10} samples are sufficient for activating effective refusal behaviors. These findings not only emphasize the importance of distributional consistency but also provide insights into the activation mechanism of safety in reasoning models.