SafeRBench: Dissecting the Reasoning Safety of Large Language Models

TL;DR

SafeRBench introduces an end-to-end evaluation framework for reasoning safety in large language models, analyzing safety risks throughout the reasoning process and providing detailed metrics to identify safety breakdowns.

Contribution

It is the first comprehensive framework to assess safety during the entire reasoning process of LRMs, including risk stratification and trace analysis.

Findings

Enabling thinking modes improves safety in mid-sized models.

Larger models show increased risks due to an always-help tendency.

The framework provides detailed metrics for safety risk assessment.

Abstract

Large Reasoning Models (LRMs) have significantly improved problem-solving through explicit Chain-of-Thought (CoT) reasoning. However, this capability creates a Safety-Helpfulness Paradox: the reasoning process itself can be misused to justify harmful actions or conceal malicious intent behind lengthy intermediate steps. Most existing benchmarks only check the final output, missing how risks evolve, or ``drift'', during the model's internal reasoning. To address this, we propose SafeRBench, the first framework to evaluate LRM safety end-to-end, from the initial input to the reasoning trace and final answer. Our approach introduces: (i) a Risk Stratification Probing that uses specific risk levels to stress-test safety boundaries beyond simple topics; (ii) Micro-Thought Analysis, a new chunking method that segments traces to pinpoint exactly where safety alignment breaks down; and (iii) a comprehensive suite of 10 fine-grained metrics that, for the first time, jointly measure a model's Risk Exposure (e.g., risk level, execution feasibility) and Safety Awareness (e.g., intent awareness). Experiments on 19 LRMs reveal that while enabling Thinking modes improves safety in mid-sized models, it paradoxically increases actionable risks in larger models due to a strong always-help tendency.