Breaking the Safety-Capability Tradeoff: Reinforcement Learning with Verifiable Rewards Maintains Safety Guardrails in LLMs

TL;DR

This paper demonstrates that reinforcement learning with verifiable rewards (RLVR) can improve reasoning in large language models while maintaining safety, challenging the common belief of an unavoidable safety-capability tradeoff.

Contribution

It provides the first theoretical and empirical analysis showing RLVR can enhance reasoning and safety simultaneously in LLMs, with bounds on safety drift and extensive benchmark validation.

Findings

RLVR can improve reasoning capabilities in LLMs.

RLVR maintains or enhances safety guardrails in adversarial benchmarks.

Theoretical bounds show conditions where safety degradation is eliminated.

Abstract

Fine-tuning large language models (LLMs) for downstream tasks typically exhibit a fundamental safety-capability tradeoff, where improving task performance degrades safety alignment even on benign datasets. This degradation persists across standard approaches including supervised finetuning (SFT) and reinforcement learning from human feedback (RLHF). While reinforcement learning with verifiable rewards (RLVR) has emerged as a promising alternative that optimizes models on objectively measurable tasks, its safety implications remain unexplored. We present the first comprehensive theoretical and empirical analysis of safety properties in RLVR. Theoretically, we derive upper bounds on safety drift under KL-constrained optimization and prove conditions under which safety degradation is eliminated. Empirically, we conduct extensive experiments across five adversarial safety benchmarks, demonstrating that RLVR can simultaneously enhance reasoning capabilities while maintaining or improving safety guardrails. Our comprehensive ablation studies examine the effects of optimization algorithms, model scale, and task domains. Our findings challenge the prevailing assumption of an inevitable safety capability trade-off, and establish that a specific training methodology can achieve both objectives simultaneously, providing insights for the safe deployment of reasoning-capable LLMs.