Less is More: Selective Reflection for Compatible and Efficient Knowledge Distillation in Large Language Models

TL;DR

This paper introduces Selective Reflection Distillation (SRD), a data curation framework that improves large language model distillation by selecting high-quality, compatible training data, leading to better performance and reduced training costs.

Contribution

SRD is a novel, plug-and-play data curation method that enhances knowledge distillation by systematically selecting and scheduling training data based on model reflections and difficulty.

Findings

SRD improves distilled model performance across various benchmarks.

SRD reduces training runtime by up to 39%.

SRD enhances sample efficiency without altering existing KD algorithms.

Abstract

Knowledge Distillation (KD) is a fundamental technique for compressing large language models (LLMs) into compact, efficient student models. However, existing white-box KD methods mainly focus on balancing ground truth and student-generated responses while overlooking two critical factors: training data quality and student-model compatibility. To address these limitations, we propose Selective Reflection Distillation (SRD), a novel data curation framework that leverages reflections from student models to systematically refine training data. SRD dynamically evaluates and selects prompt-response pairs by comparing ground truth data with student model outputs, selectively curating high-quality, student-compatible training instances through automated ranking based on difficulty. Furthermore, after selecting the training data, a curriculum scheduling strategy is employed to incrementally introduce these curated subsets into the distillation process at fixed intervals. As a plug-and-play enhancement, SRD consistently improves distillation outcomes across diverse white-box KD approaches and model architectures, as well as decreases computational cost significantly during KD training. Experiments on a range of language model benchmarks demonstrate SRD's consistent improvements in distilled model performance, as well as a reduction in training runtime by up to 39%, under diverse KD methods and model families. Notably, SRD operates as a plug-and-play module, enhancing sample efficiency without modifying underlying KD algorithms. Our findings highlight that data quality and compatibility are pivotal to effective and efficient distillation of LLMs, and SRD provides a principled framework to achieve both. This work advances the understanding of data-centric factors in KD and offers practical insights for enhancing the capability and efficiency of compressed LLMs.