How Does Prefix Matter in Reasoning Model Tuning?

TL;DR

This paper investigates how including reasoning and safety prefixes in fine-tuning datasets influences model performance, finding that prefixes improve safety and reasoning but may slightly hinder factuality and coding tasks.

Contribution

The study demonstrates that prefix conditioning acts as an implicit alignment method, enhancing safety and reasoning in models through systematic prefix inclusion during fine-tuning.

Findings

Prefix-conditioned SFT improves safety metrics by up to 6%.

Prefix inclusion enhances reasoning accuracy by 7% on GSM8K.

Prefix tokens like 'revised' stabilize reasoning trajectories.

Abstract

Recent alignment studies commonly remove introductory boilerplate phrases from supervised fine-tuning (SFT) datasets. This work challenges that assumption. We hypothesize that safety- and reasoning-oriented prefix sentences serve as lightweight alignment signals that can guide model decoding toward safer and more coherent responses. To examine this, we fine-tune three R1 series models across three core model capabilities: reasoning (mathematics, coding), safety, and factuality, systematically varying prefix inclusion from 0% to 100%. Results show that prefix-conditioned SFT improves both safety and reasoning performance, yielding up to +6% higher Safe@1 accuracy on adversarial benchmarks (WildJailbreak, StrongReject) and +7% improvement on GSM8K reasoning. However, factuality and coding tasks show marginal or negative effects, indicating that prefix-induced narrowing of the search space benefits structured reasoning. Token-level loss analysis further reveals that prefix tokens such as "revised" and "logically" incur higher gradient magnitudes, acting as alignment anchors that stabilize reasoning trajectories. Our findings suggest that prefix conditioning offers a scalable and interpretable mechanism for improving reasoning safety, serving as an implicit form of alignment that complements traditional reward-based methods.