Dynamic Contrastive Knowledge Distillation for Efficient Image Restoration

TL;DR

This paper introduces a dynamic contrastive knowledge distillation framework for image restoration that adapts to the student's learning state and leverages distribution alignment, significantly improving performance over existing methods.

Contribution

The proposed DCKD framework dynamically adjusts the solution space using contrastive learning and aligns pixel-level distributions, offering a structure-agnostic approach for improved image restoration.

Findings

DCKD outperforms state-of-the-art KD methods across tasks.

It effectively perceives the student's learning state.

It is compatible with various backbones.

Abstract

Knowledge distillation (KD) is a valuable yet challenging approach that enhances a compact student network by learning from a high-performance but cumbersome teacher model. However, previous KD methods for image restoration overlook the state of the student during the distillation, adopting a fixed solution space that limits the capability of KD. Additionally, relying solely on L1-type loss struggles to leverage the distribution information of images. In this work, we propose a novel dynamic contrastive knowledge distillation (DCKD) framework for image restoration. Specifically, we introduce dynamic contrastive regularization to perceive the student's learning state and dynamically adjust the distilled solution space using contrastive learning. Additionally, we also propose a distribution mapping module to extract and align the pixel-level category distribution of the teacher and student models. Note that the proposed DCKD is a structure-agnostic distillation framework, which can adapt to different backbones and can be combined with methods that optimize upper-bound constraints to further enhance model performance. Extensive experiments demonstrate that DCKD significantly outperforms the state-of-the-art KD methods across various image restoration tasks and backbones.