HyperSparse Neural Networks: Shifting Exploration to Exploitation through Adaptive Regularization

TL;DR

This paper introduces HyperSparse, a novel adaptive regularization method for training sparse neural networks that effectively compresses models while maintaining high performance, especially at extreme sparsity levels.

Contribution

The paper proposes Adaptive Regularized Training (ART) with HyperSparse loss, enabling iterative weight shrinking and high sparsity compression while preserving model accuracy.

Findings

Achieves up to 99.8% sparsity with notable performance gains.

Outperforms existing sparsification methods on CIFAR and TinyImageNet.

Provides insights into weight patterns at high magnitudes.

Abstract

Sparse neural networks are a key factor in developing resource-efficient machine learning applications. We propose the novel and powerful sparse learning method Adaptive Regularized Training (ART) to compress dense into sparse networks. Instead of the commonly used binary mask during training to reduce the number of model weights, we inherently shrink weights close to zero in an iterative manner with increasing weight regularization. Our method compresses the pre-trained model knowledge into the weights of highest magnitude. Therefore, we introduce a novel regularization loss named HyperSparse that exploits the highest weights while conserving the ability of weight exploration. Extensive experiments on CIFAR and TinyImageNet show that our method leads to notable performance gains compared to other sparsification methods, especially in extremely high sparsity regimes up to 99.8 percent model sparsity. Additional investigations provide new insights into the patterns that are encoded in weights with high magnitudes.