CES-KD: Curriculum-based Expert Selection for Guided Knowledge Distillation

TL;DR

CES-KD introduces a curriculum-based expert selection method that gradually guides a student network using stratified teachers based on data difficulty, improving knowledge distillation performance across multiple datasets and architectures.

Contribution

The paper proposes CES-KD, a novel curriculum-driven expert selection technique for knowledge distillation that dynamically chooses teachers based on data difficulty to better bridge the capacity gap.

Findings

CES-KD improves accuracy on various architectures.

The method is effective across multiple datasets.

Gradual teacher selection enhances student learning.

Abstract

Knowledge distillation (KD) is an effective tool for compressing deep classification models for edge devices. However, the performance of KD is affected by the large capacity gap between the teacher and student networks. Recent methods have resorted to a multiple teacher assistant (TA) setting for KD, which sequentially decreases the size of the teacher model to relatively bridge the size gap between these models. This paper proposes a new technique called Curriculum Expert Selection for Knowledge Distillation (CES-KD) to efficiently enhance the learning of a compact student under the capacity gap problem. This technique is built upon the hypothesis that a student network should be guided gradually using stratified teaching curriculum as it learns easy (hard) data samples better and faster from a lower (higher) capacity teacher network. Specifically, our method is a gradual TA-based KD technique that selects a single teacher per input image based on a curriculum driven by the difficulty in classifying the image. In this work, we empirically verify our hypothesis and rigorously experiment with CIFAR-10, CIFAR-100, CINIC-10, and ImageNet datasets and show improved accuracy on VGG-like models, ResNets, and WideResNets architectures.