REDistill: Robust Estimator Distillation for Balancing Robustness and Efficiency

TL;DR

REDistill introduces a robust distillation framework using power divergence loss to effectively handle noisy teacher outputs, improving student model accuracy without extensive hyper-parameter tuning across various architectures.

Contribution

It proposes a principled robust estimator distillation method based on power divergence, enhancing knowledge distillation robustness and generalization.

Findings

Consistently improves student accuracy on CIFAR-100 and ImageNet-1k.

Requires no hyper-parameter tuning for different teacher-student pairs.

Seamlessly integrates into existing KD pipelines with negligible overhead.

Abstract

Knowledge Distillation (KD) transfers knowledge from a large teacher model to a smaller student by aligning their predictive distributions. However, conventional KD formulations - typically based on Kullback-Leibler divergence - assume that the teacher provides reliable soft targets. In practice, teacher predictions are often noisy or overconfident, and existing correction-based approaches rely on ad-hoc heuristics and extensive hyper-parameter tuning, which hinders generalization. We introduce REDistill (Robust Estimator Distillation), a simple yet principled framework grounded in robust statistics. REDistill replaces the standard KD objective with a power divergence loss, a generalization of KL divergence that adaptively downweights unreliable teacher output while preserving informative logit relationships. This formulation provides a unified and interpretable treatment of teacher noise, requires only logits, integrates seamlessly into existing KD pipelines, and incurs negligible computational overhead. Extensive experiments on CIFAR-100 and ImageNet-1k demonstrate that REDistill consistently improves student accuracy in diverse teacher-student architectures. Remarkably, it achieves these gains without model-specific hyper-parameter tuning, underscoring its robustness and strong generalization to unseen teacher-student pairs.