Discriminative and Consistent Representation Distillation

TL;DR

This paper introduces Discriminative and Consistent Distillation (DCD), a novel knowledge distillation method that combines contrastive loss with consistency regularization, achieving state-of-the-art results and better generalization across datasets.

Contribution

DCD innovatively integrates contrastive learning with adaptive parameters and consistency regularization for improved knowledge distillation.

Findings

DCD outperforms existing methods on CIFAR-100 and ImageNet.

Student models sometimes surpass teacher accuracy.

DCD's representations generalize well to other datasets.

Abstract

Knowledge Distillation (KD) aims to transfer knowledge from a large teacher model to a smaller student model. While contrastive learning has shown promise in self-supervised learning by creating discriminative representations, its application in knowledge distillation remains limited and focuses primarily on discrimination, neglecting the structural relationships captured by the teacher model. To address this limitation, we propose Discriminative and Consistent Distillation (DCD), which employs a contrastive loss along with a consistency regularization to minimize the discrepancy between the distributions of teacher and student representations. Our method introduces learnable temperature and bias parameters that adapt during training to balance these complementary objectives, replacing the fixed hyperparameters commonly used in contrastive learning approaches. Through extensive experiments on CIFAR-100 and ImageNet ILSVRC-2012, we demonstrate that DCD achieves state-of-the-art performance, with the student model sometimes surpassing the teacher's accuracy. Furthermore, we show that DCD's learned representations exhibit superior cross-dataset generalization when transferred to Tiny ImageNet and STL-10.