Semantic Relation Preserving Knowledge Distillation for Image-to-Image Translation

TL;DR

This paper introduces a novel knowledge distillation method that preserves semantic relations in GAN-based image-to-image translation, significantly reducing model size while maintaining high translation quality.

Contribution

It proposes a semantic relation preserving matrix for knowledge distillation tailored to image-to-image translation tasks, improving model compression effectiveness.

Findings

Achieves high-quality image translation with smaller models

Effective across multiple datasets and model pairs

Outperforms existing compression methods in this domain

Abstract

Generative adversarial networks (GANs) have shown significant potential in modeling high dimensional distributions of image data, especially on image-to-image translation tasks. However, due to the complexity of these tasks, state-of-the-art models often contain a tremendous amount of parameters, which results in large model size and long inference time. In this work, we propose a novel method to address this problem by applying knowledge distillation together with distillation of a semantic relation preserving matrix. This matrix, derived from the teacher's feature encoding, helps the student model learn better semantic relations. In contrast to existing compression methods designed for classification tasks, our proposed method adapts well to the image-to-image translation task on GANs. Experiments conducted on 5 different datasets and 3 different pairs of teacher and student models provide strong evidence that our methods achieve impressive results both qualitatively and quantitatively.