How to Fine-Tune a Reasoning Model? A Teacher-Student Cooperation Framework to Synthesize Student-Consistent SFT Data

TL;DR

This paper introduces TESSY, a framework that improves fine-tuning of reasoning models by synthesizing data that balances teacher expertise with student style, leading to better reasoning performance.

Contribution

The paper proposes TESSY, a novel teacher-student cooperation method for data synthesis that enhances reasoning model fine-tuning by maintaining stylistic consistency.

Findings

TESSY outperforms traditional teacher-generated data in reasoning tasks.

Fine-tuning with TESSY data improves performance on code generation benchmarks.

Teacher-student interleaving reduces stylistic divergence and boosts reasoning capabilities.

Abstract

A widely adopted strategy for model enhancement is to use synthetic data generated by a stronger model for supervised fine-tuning (SFT). However, for emerging reasoning models like Qwen3-8B, this approach often fails to improve reasoning capabilities and can even lead to a substantial drop in performance. In this work, we identify substantial stylistic divergence between teacher generated data and the distribution of student as a major factor impacting SFT. To bridge this gap, we propose a Teacher-Student Cooperation Data Synthesis framework (TESSY), which interleaves teacher and student models to alternately generate style and non-style tokens. Consequently, TESSY produces synthetic sequences that inherit the advanced reasoning capabilities of the teacher while maintaining stylistic consistency with the distribution of the student. In experiments on code generation using GPT-OSS-120B as the teacher, fine-tuning Qwen3-8B on teacher-generated data leads to performance drops of 3.25% on LiveCodeBench-Pro and 10.02% on OJBench, whereas TESSY achieves improvements of 11.25% and 6.68%.