ONG: One-Shot NMF-based Gradient Masking for Efficient Model Sparsification

TL;DR

ONG introduces a one-shot pruning method using NMF to identify important weights and gradient masking to maintain sparsity during training, enabling efficient DNN compression with competitive performance.

Contribution

This paper presents ONG, a novel one-shot sparsification technique combining NMF-based weight analysis and gradient masking for effective model compression.

Findings

Achieves comparable or better accuracy at various sparsity levels.

Maintains structural integrity of pruned models.

Operates efficiently with a one-shot pruning approach.

Abstract

Deep Neural Networks (DNNs) have achieved remarkable success but their large size poses deployment challenges. While various pruning techniques exist, many involve complex iterative processes, specialized criteria, or struggle to maintain sparsity effectively during training. We introduce ONG (One-shot NMF-based Gradient Masking), a novel sparsification strategy that identifies salient weight structures using Non-negative Matrix Factorization (NMF) for one-shot pruning at the outset of training. Subsequently, ONG employs a precise gradient masking mechanism to ensure that only unpruned weights are updated, strictly preserving the target sparsity throughout the training phase. We integrate ONG into the BIMP comparative framework and evaluate it on CIFAR-10 and CIFAR-100 with ResNet56, ResNet34, and ResNet18 against established stable sparsification methods. Our experiments demonstrate ONG's ability to achieve comparable or superior performance at various sparsity levels while maintaining structural integrity post-pruning and offering a clear mechanism for targeting desired sparsities.