Layer-wise Model Pruning based on Mutual Information

TL;DR

This paper introduces a layer-wise model pruning method based on mutual information that improves speed and accuracy over traditional weight-based pruning by leveraging global training signals and dense representations.

Contribution

It proposes a novel pruning strategy that operates globally using mutual information, leading to more efficient and effective model compression.

Findings

Achieves greater speedup than weight-based pruning methods.

Maintains higher performance at the same sparsity levels.

Operates from a top-down, global perspective for pruning.

Abstract

The proposed pruning strategy offers merits over weight-based pruning techniques: (1) it avoids irregular memory access since representations and matrices can be squeezed into their smaller but dense counterparts, leading to greater speedup; (2) in a manner of top-down pruning, the proposed method operates from a more global perspective based on training signals in the top layer, and prunes each layer by propagating the effect of global signals through layers, leading to better performances at the same sparsity level. Extensive experiments show that at the same sparsity level, the proposed strategy offers both greater speedup and higher performances than weight-based pruning methods (e.g., magnitude pruning, movement pruning).