LOss-Based SensiTivity rEgulaRization: towards deep sparse neural networks

TL;DR

LOSTER is a novel regularization method that promotes sparsity in neural networks by shrinking low-sensitivity parameters during training, enabling efficient pruning without pre-training or rewinding, and achieving competitive compression.

Contribution

It introduces a sensitivity-based regularization technique that trains sparse neural networks from scratch with minimal overhead, unlike existing methods requiring pre-training.

Findings

Achieves competitive compression ratios across multiple architectures.

Enables training sparse networks from scratch without pre-training.

Maintains minimal computational overhead during training.

Abstract

LOBSTER (LOss-Based SensiTivity rEgulaRization) is a method for training neural networks having a sparse topology. Let the sensitivity of a network parameter be the variation of the loss function with respect to the variation of the parameter. Parameters with low sensitivity, i.e. having little impact on the loss when perturbed, are shrunk and then pruned to sparsify the network. Our method allows to train a network from scratch, i.e. without preliminary learning or rewinding. Experiments on multiple architectures and datasets show competitive compression ratios with minimal computational overhead.