Class-Discriminative CNN Compression

TL;DR

This paper introduces class-discriminative CNN compression (CDC), combining hierarchical pruning and discriminant-based distillation to improve CNN efficiency while maintaining high accuracy, validated on CIFAR and ImageNet datasets.

Contribution

The paper proposes a novel class-discriminative compression framework that integrates hierarchical pruning with discriminant analysis-based distillation, enhancing CNN compression effectiveness.

Findings

CDC outperforms state-of-the-art methods on CIFAR and ImageNet.

Layer-adaptive pruning aligns with CNN semantic processing.

Discriminant component analysis improves hidden layer separability.

Abstract

Compressing convolutional neural networks (CNNs) by pruning and distillation has received ever-increasing focus in the community. In particular, designing a class-discrimination based approach would be desired as it fits seamlessly with the CNNs training objective. In this paper, we propose class-discriminative compression (CDC), which injects class discrimination in both pruning and distillation to facilitate the CNNs training goal. We first study the effectiveness of a group of discriminant functions for channel pruning, where we include well-known single-variate binary-class statistics like Student's T-Test in our study via an intuitive generalization. We then propose a novel layer-adaptive hierarchical pruning approach, where we use a coarse class discrimination scheme for early layers and a fine one for later layers. This method naturally accords with the fact that CNNs process coarse semantics in the early layers and extract fine concepts at the later. Moreover, we leverage discriminant component analysis (DCA) to distill knowledge of intermediate representations in a subspace with rich discriminative information, which enhances hidden layers' linear separability and classification accuracy of the student. Combining pruning and distillation, CDC is evaluated on CIFAR and ILSVRC 2012, where we consistently outperform the state-of-the-art results.