On-Policy Consistency Training Improves LLM Safety with Minimal Capability Degradation

TL;DR

On-Policy Consistency Training (OPCT) enhances Large Language Model safety across multiple axes with minimal capability loss by training models on their own responses conditioned on contrastive prompts.

Contribution

The paper introduces OPCT, a novel on-policy consistency training method that outperforms traditional supervised fine-tuning in safety and robustness without significant capability degradation.

Findings

OPCT nearly halves sycophancy rate compared to baseline.

OPCT achieves near 99% jailbreak defense success on held-out behaviors.

OPCT avoids capability regressions seen in supervised fine-tuning.

Abstract

Aligned models can misbehave in several ways: they are often sycophantic, fall victim to jailbreaks, or fail to include appropriate safety warnings. Consistency training is a promising new alignment paradigm to mitigate such failures by training invariants into the model using contrastive input pairs. Existing consistency training procedures generate the supervision signal once, offline, and use supervised fine-tuning (SFT) to update the model. Unfortunately, the resulting models tend to merely memorize the surface forms of the training distribution and thus generalize poorly and regress in their capabilities. We introduce On-Policy Consistency Training (OPCT), a new consistency training approach where the objective is computed over the model's own responses to prompts, supervised by itself conditioned on corresponding contrastive prompts. We evaluate OPCT on three safety axes: sycophancy, jailbreaking, and safety awareness. Across three model families, OPCT outperforms its SFT counterpart on all safety desiderata. It nearly halves the sycophancy rate relative to baseline (8.1% vs. 15.4%, compared to 11.2% for SFT). Under an adaptive per-target attacker, OPCT holds jailbreak defense success near 99% on held-out jailbreak behaviors, whereas SFT achieves 87% on average. On safety awareness, OPCT outperforms SFT in two out of three models, and matches it on the other. OPCT also largely avoids the capability regressions that SFT induces, such as a 28-point drop on MATH-500. Our results suggest that consistency training is best implemented as OPCT rather than as SFT, especially when generalization beyond the training distribution is desired.