Magnitude Attention-based Dynamic Pruning

TL;DR

This paper introduces MAP, a dynamic pruning method that uses magnitude attention during training to adaptively identify and update important weights, resulting in efficient sparse models with performance comparable or superior to dense models.

Contribution

The novel MAP approach applies importance-based magnitude attention throughout training, enabling dynamic exploration and exploitation of sparse structures for improved pruning effectiveness.

Findings

Achieves comparable or better accuracy than dense models on CIFAR-10/100 and ImageNet.

Effectively shifts from exploration to exploitation during training.

Outperforms previous pruning methods in experiments.

Abstract

Existing pruning methods utilize the importance of each weight based on specified criteria only when searching for a sparse structure but do not utilize it during training. In this work, we propose a novel approach - \textbf{M}agnitude \textbf{A}ttention-based Dynamic \textbf{P}runing (MAP) method, which applies the importance of weights throughout both the forward and backward paths to explore sparse model structures dynamically. Magnitude attention is defined based on the magnitude of weights as continuous real-valued numbers enabling a seamless transition from a redundant to an effective sparse network by promoting efficient exploration. Additionally, the attention mechanism ensures more effective updates for important layers within the sparse network. In later stages of training, our approach shifts from exploration to exploitation, exclusively updating the sparse model composed of crucial weights based on the explored structure, resulting in pruned models that not only achieve performance comparable to dense models but also outperform previous pruning methods on CIFAR-10/100 and ImageNet.