Distilling the Essence: Efficient Reasoning Distillation via Sequence Truncation

TL;DR

This paper proposes a sequence truncation method for efficient knowledge distillation in reasoning models, showing that training on half the sequence length retains most performance while reducing computational costs.

Contribution

It introduces a truncation protocol that identifies optimal sequence lengths for distillation, significantly reducing resource usage with minimal performance loss.

Findings

Training on 50% of sequence tokens retains ~91% of performance.

Sequence truncation reduces memory and FLOPs by about 50%.

Selective distillation over chain-of-thought tokens is effective.

Abstract

Distilling the capabilities from a large reasoning model (LRM) to a smaller student model often involves training on substantial amounts of reasoning data. However, knowledge distillation (KD) over lengthy sequences with prompt (P), chain-of-thought (CoT), and answer (A) sections makes the process computationally expensive. In this work, we investigate how the allocation of supervision across different sections (P, CoT, A) affects student performance. Our analysis shows that selective KD over only the CoT tokens can be effective when the prompt and answer information is encompassed by it. Building on this insight, we establish a truncation protocol to quantify computation-quality tradeoffs as a function of sequence length. We observe that beyond a specific length, longer training sequences provide marginal returns for downstream performance but require substantially higher memory and FLOPs. To this end, training on only the first $50\%$ of tokens of every training sequence can retain, on average, $\approx91\%$ of full-sequence performance on math benchmarks while reducing training time, memory usage, and FLOPs by about $50\%$ each. Codes are available at https://github.com/weiruichen01/distilling-the-essence.