Learning from Diverse Reasoning Paths with Routing and Collaboration

TL;DR

This paper introduces QR-Distill, a novel knowledge distillation method that uses quality filtering, conditional routing, and peer teaching to improve reasoning in compact language models by effectively capturing diverse reasoning paths.

Contribution

It presents QR-Distill, a new approach combining path quality filtering, dynamic routing, and cooperative peer teaching for better knowledge transfer from large to small language models.

Findings

QR-Distill outperforms traditional distillation methods.

Each component (quality filtering, routing, peer teaching) significantly improves performance.

Ablation studies confirm the effectiveness of all proposed components.

Abstract

Advances in large language models (LLMs) significantly enhance reasoning capabilities but their deployment is restricted in resource-constrained scenarios. Knowledge distillation addresses this by transferring knowledge from powerful teacher models to compact and transparent students. However, effectively capturing the teacher's comprehensive reasoning is challenging due to conventional token-level supervision's limited scope. Using multiple reasoning paths per query alleviates this problem, but treating each path identically is suboptimal as paths vary widely in quality and suitability across tasks and models. We propose Quality-filtered Routing with Cooperative Distillation (QR-Distill), combining path quality filtering, conditional routing, and cooperative peer teaching. First, quality filtering retains only correct reasoning paths scored by an LLM-based evaluation. Second, conditional routing dynamically assigns paths tailored to each student's current learning state. Finally, cooperative peer teaching enables students to mutually distill diverse insights, addressing knowledge gaps and biases toward specific reasoning styles. Experiments demonstrate QR-Distill's superiority over traditional single- and multi-path distillation methods. Ablation studies further highlight the importance of each component including quality filtering, conditional routing, and peer teaching in effective knowledge transfer. Our code is available at https://github.com/LzyFischer/Distill.