Mitigating the Safety-utility Trade-off in LLM Alignment via Adaptive Safe Context Learning

TL;DR

This paper introduces the Adaptive Safe Context Learning framework to improve safety and reasoning in large language models by enabling autonomous safety rule consultation and rebalancing rule usage during training.

Contribution

The paper proposes a novel multi-turn tool-use approach and an inverse frequency policy optimization to better balance safety and utility in LLM alignment.

Findings

Enhanced reasoning performance over baselines

Effective mitigation of safety-utility trade-off

Autonomous safety rule consultation improves reasoning

Abstract

While reasoning models have achieved remarkable success in complex reasoning tasks, their increasing power necessitates stringent safety measures. For safety alignment, the core challenge lies in the inherent trade-off between safety and utility. However, prevailing alignment strategies typically construct CoT training data with explicit safety rules via context distillation. This approach inadvertently limits reasoning capabilities by creating a rigid association between rule memorization and refusal. To mitigate the safety-utility trade-off, we propose the Adaptive Safe Context Learning (ASCL) framework to improve the reasoning given proper context. ASCL formulates safety alignment as a multi-turn tool-use process, empowering the model to autonomously decide when to consult safety rules and how to generate the ongoing reasoning. Furthermore, to counteract the preference for rule consultation during RL, we introduce Inverse Frequency Policy Optimization (IFPO) to rebalance advantage estimates. By decoupling rule retrieval and subsequent reasoning, our method achieves higher overall performance compared to baselines.