# MICIK: MIning Cross-Layer Inherent Similarity Knowledge for Deep Model Compression

## TL;DR

MICIK is a novel deep model compression framework that exploits cross-layer redundancy and inherent similarity knowledge to achieve significant parameter reduction without accuracy loss.

## Contribution

It introduces a holistic approach to model compression by mining cross-layer inherent similarities and common components, surpassing existing methods in compression rate.

## Key findings

- 16X parameter reduction on VGG-16
- 6X parameter reduction on GoogLeNet
- No accuracy loss achieved

## Abstract

State-of-the-art deep model compression methods exploit the low-rank approximation and sparsity pruning to remove redundant parameters from a learned hidden layer. However, they process each hidden layer individually while neglecting the common components across layers, and thus are not able to fully exploit the potential redundancy space for compression. To solve the above problem and enable further compression of a model, removing the cross-layer redundancy and mining the layer-wise inheritance knowledge is necessary. In this paper, we introduce a holistic model compression framework, namely MIning Cross-layer Inherent similarity Knowledge (MICIK), to fully excavate the potential redundancy space. The proposed MICIK framework simultaneously, (1) learns the common and unique weight components across deep neural network layers to increase compression rate; (2) preserves the inherent similarity knowledge of nearby layers and distant layers to minimize the accuracy loss and (3) can be complementary to other existing compression techniques such as knowledge distillation. Extensive experiments on large-scale convolutional neural networks demonstrate that MICIK is superior over state-of-the-art model compression approaches with 16X parameter reduction on VGG-16 and 6X on GoogLeNet, all without accuracy loss.