Explain in Your Own Words: Improving Reasoning via Token-Selective Dual Knowledge Distillation

TL;DR

This paper introduces Token-Selective Dual Knowledge Distillation (TSD-KD), a novel framework that enhances reasoning in smaller models by focusing on important tokens and combining indirect and direct feedback mechanisms, leading to state-of-the-art results.

Contribution

TSD-KD is a new student-centric distillation method that selectively distills tokens and uses indirect feedback, improving reasoning abilities of small models beyond existing methods.

Findings

Achieves up to 54.4% accuracy improvement on reasoning benchmarks.

Outperforms baseline and runner-up models significantly.

In some cases, student models surpass their teachers by up to 20.3%.

Abstract

Knowledge Distillation (KD) can transfer the reasoning abilities of large models to smaller ones, which can reduce the costs to generate Chain-of-Thoughts for reasoning tasks. KD methods typically ask the student to mimic the teacher's distribution over the entire output. However, a student with limited capacity can be overwhelmed by such extensive supervision causing a distribution mismatch, especially in complex reasoning tasks. We propose Token-Selective Dual Knowledge Distillation (TSD-KD), a framework for student-centric distillation. TSD-KD focuses on distilling important tokens for reasoning and encourages the student to explain reasoning in its own words. TSD-KD combines indirect and direct distillation. Indirect distillation uses a weak form of feedback based on preference ranking. The student proposes candidate responses generated on its own; the teacher re-ranks those candidates as indirect feedback without enforcing its entire distribution. Direct distillation uses distribution matching; however, it selectively distills tokens based on the relative confidence between teacher and student. Finally, we add entropy regularization to maintain the student's confidence during distillation. Overall, our method provides the student with targeted and indirect feedback to support its own reasoning process and to facilitate self-improvement. The experiments show the state-of-the-art performance of TSD-KD on 10 challenging reasoning benchmarks, outperforming the baseline and runner-up in accuracy by up to 54.4\% and 40.3\%, respectively. Notably, a student trained by TSD-KD even outperformed its own teacher model in four cases by up to 20.3\%. The source code is available at https://github.com/kmswin1/TSD-KD.