Paraphrasing Complex Network: Network Compression via Factor Transfer

TL;DR

This paper introduces a novel knowledge transfer technique called factor transfer, using convolutional modules to paraphrase and translate teacher network knowledge, resulting in improved model compression for deep neural networks.

Contribution

The paper proposes a new factor transfer method employing convolutional modules for better knowledge transfer in neural network compression, outperforming traditional methods.

Findings

Student networks with factor transfer outperform conventional knowledge transfer methods.

The paraphraser module is trained in an unsupervised manner to extract teacher factors.

The method effectively reduces model size with minimal performance loss.

Abstract

Many researchers have sought ways of model compression to reduce the size of a deep neural network (DNN) with minimal performance degradation in order to use DNNs in embedded systems. Among the model compression methods, a method called knowledge transfer is to train a student network with a stronger teacher network. In this paper, we propose a novel knowledge transfer method which uses convolutional operations to paraphrase teacher's knowledge and to translate it for the student. This is done by two convolutional modules, which are called a paraphraser and a translator. The paraphraser is trained in an unsupervised manner to extract the teacher factors which are defined as paraphrased information of the teacher network. The translator located at the student network extracts the student factors and helps to translate the teacher factors by mimicking them. We observed that our student network trained with the proposed factor transfer method outperforms the ones trained with conventional knowledge transfer methods.