EGAD: Entropy-Guided Adaptive Distillation for Token-Level Knowledge Transfer

TL;DR

EGAD introduces an entropy-guided adaptive distillation method that dynamically focuses on different tokens during training, improving knowledge transfer efficiency for large language models.

Contribution

The paper proposes a novel entropy-based adaptive distillation strategy that adjusts token focus, temperature, and architecture based on token entropy for improved model compression.

Findings

Enhanced distillation efficiency demonstrated on benchmark tasks.

Dynamic token focus improves learning from difficult tokens.

Dual-branch architecture balances logits and feature distillation.

Abstract

Large language models (LLMs) have achieved remarkable performance across diverse domains, yet their enormous computational and memory requirements hinder deployment in resource-constrained environments. Knowledge distillation offers a promising solution by transferring knowledge from a large teacher model to a smaller student model. However, existing distillation methods typically treat all tokens equally, ignoring the fact that different tokens contribute unequally to model decisions. This can lead to inefficient knowledge transfer and reduced learning effectiveness. To address this limitation, we propose an entropy-based adaptive distillation strategy that dynamically adjusts the training process at the token level. Our method leverages the teacher's output entropy to guide three aspects of distillation. Specifically, we introduce a token-level curriculum by dynamically shifting focus from low- to high-entropy tokens during training. We further adjust the distillation temperature based on token entropy to better capture teacher confidence patterns. Moreover, we employ a dual-branch architecture for efficient logits-only distillation on easy tokens and deeper feature-based distillation on difficult tokens. Extensive experiments validate the soundness and effectiveness of our method.