Distilling and Transferring Knowledge via cGAN-generated Samples for Image Classification and Regression

TL;DR

This paper introduces cGAN-KD, a unified knowledge distillation framework using cGAN-generated samples, effective for both image classification and regression, and compatible with existing KD methods.

Contribution

The paper proposes a novel cGAN-based KD framework that works for classification and regression, is architecture-insensitive, and enhances state-of-the-art performance.

Findings

cGAN-KD improves performance on CIFAR-100 and ImageNet-100

It effectively applies to image regression tasks like Steering Angle and UTKFace

Combining cGAN-KD with existing KD methods yields state-of-the-art results.

Abstract

Knowledge distillation (KD) has been actively studied for image classification tasks in deep learning, aiming to improve the performance of a student based on the knowledge from a teacher. However, applying KD in image regression with a scalar response variable has been rarely studied, and there exists no KD method applicable to both classification and regression tasks yet. Moreover, existing KD methods often require a practitioner to carefully select or adjust the teacher and student architectures, making these methods less flexible in practice. To address the above problems in a unified way, we propose a comprehensive KD framework based on cGANs, termed cGAN-KD. Fundamentally different from existing KD methods, cGAN-KD distills and transfers knowledge from a teacher model to a student model via cGAN-generated samples. This novel mechanism makes cGAN-KD suitable for both classification and regression tasks, compatible with other KD methods, and insensitive to the teacher and student architectures. An error bound for a student model trained in the cGAN-KD framework is derived in this work, providing a theory for why cGAN-KD is effective as well as guiding the practical implementation of cGAN-KD. Extensive experiments on CIFAR-100 and ImageNet-100 show that we can combine state of the art KD methods with the cGAN-KD framework to yield a new state of the art. Moreover, experiments on Steering Angle and UTKFace demonstrate the effectiveness of cGAN-KD in image regression tasks, where existing KD methods are inapplicable.