Knowledge Distillation for Image Restoration : Simultaneous Learning from Degraded and Clean Images

TL;DR

This paper introduces a novel knowledge distillation framework for image restoration that enables simultaneous learning from degraded and clean images, significantly reducing model complexity while preserving high restoration quality.

Contribution

The proposed SLKD framework uniquely employs dual teachers and dual learning strategies for effective model compression in image restoration tasks.

Findings

Achieves over 80% reduction in FLOPs and parameters.

Maintains strong restoration performance across multiple datasets.

Demonstrates effectiveness in three image restoration tasks.

Abstract

Model compression through knowledge distillation has seen extensive application in classification and segmentation tasks. However, its potential in image-to-image translation, particularly in image restoration, remains underexplored. To address this gap, we propose a Simultaneous Learning Knowledge Distillation (SLKD) framework tailored for model compression in image restoration tasks. SLKD employs a dual-teacher, single-student architecture with two distinct learning strategies: Degradation Removal Learning (DRL) and Image Reconstruction Learning (IRL), simultaneously. In DRL, the student encoder learns from Teacher A to focus on removing degradation factors, guided by a novel BRISQUE extractor. In IRL, the student decoder learns from Teacher B to reconstruct clean images, with the assistance of a proposed PIQE extractor. These strategies enable the student to learn from degraded and clean images simultaneously, ensuring high-quality compression of image restoration models. Experimental results across five datasets and three tasks demonstrate that SLKD achieves substantial reductions in FLOPs and parameters, exceeding 80\%, while maintaining strong image restoration performance.