Adversarial Network Compression

TL;DR

This paper introduces an adversarial network compression method that effectively transfers knowledge from large to small neural networks without labels, achieving superior performance and generalization across multiple datasets.

Contribution

It presents a novel adversarial training approach for neural network compression, including a regularization scheme to enhance stability and generalization.

Findings

Student networks achieve minimal accuracy loss.

Outperforms existing knowledge transfer methods.

Surpasses performance of labeled training on the same architecture.

Abstract

Neural network compression has recently received much attention due to the computational requirements of modern deep models. In this work, our objective is to transfer knowledge from a deep and accurate model to a smaller one. Our contributions are threefold: (i) we propose an adversarial network compression approach to train the small student network to mimic the large teacher, without the need for labels during training; (ii) we introduce a regularization scheme to prevent a trivially-strong discriminator without reducing the network capacity and (iii) our approach generalizes on different teacher-student models. In an extensive evaluation on five standard datasets, we show that our student has small accuracy drop, achieves better performance than other knowledge transfer approaches and it surpasses the performance of the same network trained with labels. In addition, we demonstrate state-of-the-art results compared to other compression strategies.