Compact CNN Structure Learning by Knowledge Distillation

TL;DR

This paper introduces a knowledge distillation framework with block-wise optimization to compress CNNs effectively, maintaining higher accuracy and better resource efficiency for embedded devices.

Contribution

It presents a novel customizable compression method that outperforms existing techniques in balancing accuracy and resource constraints.

Findings

Achieves up to 2x FLOPs reduction and 5.2x parameter reduction on MobileNet_v2.

Maintains or improves accuracy compared to baseline models.

Demonstrates robustness across various architectures and datasets.

Abstract

The concept of compressing deep Convolutional Neural Networks (CNNs) is essential to use limited computation, power, and memory resources on embedded devices. However, existing methods achieve this objective at the cost of a drop in inference accuracy in computer vision tasks. To address such a drawback, we propose a framework that leverages knowledge distillation along with customizable block-wise optimization to learn a lightweight CNN structure while preserving better control over the compression-performance tradeoff. Considering specific resource constraints, e.g., floating-point operations per inference (FLOPs) or model-parameters, our method results in a state of the art network compression while being capable of achieving better inference accuracy. In a comprehensive evaluation, we demonstrate that our method is effective, robust, and consistent with results over a variety of network architectures and datasets, at negligible training overhead. In particular, for the already compact network MobileNet_v2, our method offers up to 2x and 5.2x better model compression in terms of FLOPs and model-parameters, respectively, while getting 1.05% better model performance than the baseline network.