Structured Pattern Pruning Using Regularization

TL;DR

This paper introduces SPUR, a regularization-based pruning method that induces structured patterns in neural networks, preserving performance at high sparsity levels across various languages and tasks.

Contribution

SPUR is a novel regularization technique that enhances iterative magnitude pruning by preemptively creating structured patterns, improving efficiency and performance.

Findings

SPUR significantly preserves model performance at high sparsity levels.

Structured patterns in weight matrices can be effectively induced by regularization.

SPUR is resource-efficient and applicable across different languages and tasks.

Abstract

Iterative Magnitude Pruning (IMP) is a network pruning method that repeats the process of removing weights with the least magnitudes and retraining the model. When visualizing the weight matrices of language models pruned by IMP, previous research has shown that a structured pattern emerges, wherein the resulting surviving weights tend to prominently cluster in a select few rows and columns of the matrix. Though the need for further research in utilizing these structured patterns for potential performance gains has previously been indicated, it has yet to be thoroughly studied. We propose SPUR (Structured Pattern pruning Using Regularization), a novel pruning mechanism that preemptively induces structured patterns in compression by adding a regularization term to the objective function in the IMP. Our results show that SPUR can significantly preserve model performance under high sparsity settings regardless of the language or the task. Our contributions are as follows: (i) We propose SPUR, a network pruning mechanism that improves upon IMP regardless of the language or the task. (ii) We are the first to empirically verify the efficacy of "structured patterns" observed previously in pruning research. (iii) SPUR is a resource-efficient mechanism in that it does not require significant additional computations.