A Theoretical Explanation of Activation Sparsity through Flat Minima and Adversarial Robustness

TL;DR

This paper provides a theoretical explanation for activation sparsity in deep neural networks, linking it to flat minima and adversarial robustness, and demonstrates practical sparsity improvements with new modules.

Contribution

It introduces gradient sparsity as a source of activation sparsity and offers a theoretical framework connecting sparsity to flat minima and robustness in deep models.

Findings

Activation sparsity correlates with flat minima in well-trained models.

Proposed modules achieve 50% sparsity improvements on ImageNet-1k and C4.

Spectral concentration of weight matrices supports the sparsity explanation.

Abstract

A recent empirical observation (Li et al., 2022b) of activation sparsity in MLP blocks offers an opportunity to drastically reduce computation costs for free. Although having attributed it to training dynamics, existing theoretical explanations of activation sparsity are restricted to shallow networks, small training steps and special training, despite its emergence in deep models standardly trained for a large number of steps. To fill these gaps, we propose the notion of gradient sparsity as one source of activation sparsity and a theoretical explanation based on it that sees sparsity a necessary step to adversarial robustness w.r.t. hidden features and parameters, which is approximately the flatness of minima for well-learned models. The theory applies to standardly trained LayerNorm-ed MLPs, and further to Transformers or other architectures trained with weight noises. Eliminating other sources of flatness except for sparsity, we discover the phenomenon that the ratio between the largest and smallest non-zero singular values of weight matrices is small. When discussing the emergence of this spectral concentration, we use random matrix theory (RMT) as a powerful tool to analyze stochastic gradient noises. Validational experiments are conducted to verify our gradient-sparsity-based explanation. We propose two plug-and-play modules for both training and finetuning for sparsity. Experiments on ImageNet-1k and C4 demonstrate their 50% sparsity improvements, indicating further potential cost reduction in both training and inference.