The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Matter

TL;DR

This paper investigates the phenomenon of essential sparsity in large pre-trained transformers, revealing a sharp drop in performance beyond a certain sparsity level and uncovering emergent sparsification during pre-training.

Contribution

It introduces the concept of essential sparsity, demonstrates its validity across models and sparsity patterns, and uncovers emergent sparsification phenomena during BERT pre-training.

Findings

Essential sparsity exhibits a sharp performance decline beyond a certain sparsity threshold.

Emergent abrupt sparsification occurs during BERT pre-training after specific iterations.

Self-supervised learning enhances emergent sparsification compared to supervised learning.

Abstract

Large pre-trained transformers are show-stealer in modern-day deep learning, and it becomes crucial to comprehend the parsimonious patterns that exist within them as they grow in scale. With exploding parameter counts, Lottery Ticket Hypothesis (LTH) and its variants, have lost their pragmatism in sparsifying them due to high computation and memory bottleneck of repetitive train-prune-retrain routine of iterative magnitude pruning (IMP) which worsens with increasing model size. This paper comprehensively studies induced sparse patterns across multiple large pre-trained vision and language transformers. We propose the existence of -- essential sparsity defined with a sharp dropping point beyond which the performance declines much faster w.r.t the rise of sparsity level, when we directly remove weights with the smallest magnitudes in one-shot without re-training. We also find essential sparsity to hold valid for N:M sparsity patterns as well as on modern-scale large language models (Vicuna-7B). We also present an intriguing emerging phenomenon of abrupt sparsification during the pre-training of BERT, i.e., BERT suddenly becomes heavily sparse in pre-training after certain iterations. Moreover, our observations also indicate a counter-intuitive finding that BERT trained with a larger amount of pre-training data tends to have a better ability to condense knowledge in comparatively relatively fewer parameters. Lastly, we investigate the effect of the pre-training loss on essential sparsity and discover that self-supervised learning (SSL) objectives trigger stronger emergent sparsification properties than supervised learning (SL). Our codes are available at \url{https://github.com/VITA-Group/essential_sparsity}.