Contrastive Representation Distillation via Multi-Scale Feature Decoupling

TL;DR

This paper introduces MSDCRD, a novel distillation framework that decouples features into multi-scale local components and uses contrastive learning for efficient, high-performance knowledge transfer without external memory.

Contribution

It proposes a model-agnostic method that decouples features into multi-scale local parts and employs contrastive losses, improving distillation efficiency and effectiveness.

Findings

Achieves superior performance in homogeneous and heterogeneous settings.

Eliminates the need for external memory buffers.

Demonstrates strong generalization across architectures.

Abstract

Knowledge distillation enhances the performance of compact student networks by transferring knowledge from more powerful teacher networks without introducing additional parameters. In the feature space, local regions within an individual global feature encode distinct yet interdependent semantic information. Previous feature-based distillation methods mainly emphasize global feature alignment while neglecting the decoupling of local regions within an individual global feature, which often results in semantic confusion and suboptimal performance. Moreover, conventional contrastive representation distillation suffers from low efficiency due to its reliance on a large memory buffer to store feature samples. To address these limitations, this work proposes MSDCRD, a model-agnostic distillation framework that systematically decouples global features into multi-scale local features and leverages the resulting semantically rich feature samples with tailored sample-wise and feature-wise contrastive losses. This design enables efficient distillation using only a single batch, eliminating the dependence on external memory. Extensive experiments demonstrate that MSDCRD achieves superior performance not only in homogeneous teacher-student settings but also in heterogeneous architectures where feature discrepancies are more pronounced, highlighting its strong generalization capability.