Long-Short Chain-of-Thought Mixture Supervised Fine-Tuning Eliciting Efficient Reasoning in Large Language Models

TL;DR

This paper introduces LS-Mixture SFT, a fine-tuning method that combines long and short reasoning datasets to improve accuracy and reduce verbosity in large language models' reasoning processes.

Contribution

The paper proposes a novel fine-tuning approach that mitigates overthinking in reasoning models by mixing long and short reasoning datasets, enhancing efficiency and accuracy.

Findings

Achieved 2.3% average accuracy improvement across benchmarks.

Reduced model response length by approximately 47.61%.

Enabled reasoning capabilities without inheriting overthinking from teacher models.

Abstract

Recent advances in large language models have demonstrated that Supervised Fine-Tuning (SFT) with Chain-of-Thought (CoT) reasoning data distilled from large reasoning models (e.g., DeepSeek R1) can effectively transfer reasoning capabilities to non-reasoning models. However, models fine-tuned with this approach inherit the "overthinking" problem from teacher models, producing verbose and redundant reasoning chains during inference. To address this challenge, we propose Long-Short Chain-of-Thought Mixture Supervised Fine-Tuning (LS-Mixture SFT), which combines long CoT reasoning dataset with their short counterparts obtained through structure-preserved rewriting. Our experiments demonstrate that models trained using the LS-Mixture SFT method, compared to those trained with direct SFT, achieved an average accuracy improvement of 2.3% across various benchmarks while substantially reducing model response length by approximately 47.61%. This work offers an approach to endow non-reasoning models with reasoning capabilities through supervised fine-tuning while avoiding the inherent overthinking problems inherited from teacher models, thereby enabling efficient reasoning in the fine-tuned models.