A Novel Sparse Regularizer

TL;DR

This paper introduces a new entropy-based regularizer that considers the spatial arrangement of weights, leading to significant sparsity improvements in neural network training.

Contribution

A novel entropy-based regularizer that differs from traditional norm-based methods by focusing on weight spatial distribution, offering simplicity and efficiency.

Findings

Achieves roughly tenfold reduction in nonzero parameters for LeNet300 on MNIST.

Is differentiable, scalable, and easy to implement in parallel.

Provides better sparsity-accuracy trade-offs compared to existing regularizers.

Abstract

$L_p$-norm regularization schemes such as $L_0$, $L_1$, and $L_2$-norm regularization and $L_p$-norm-based regularization techniques such as weight decay, LASSO, and elastic net compute a quantity which depends on model weights considered in isolation from one another. This paper introduces a regularizer based on minimizing a novel measure of entropy applied to the model during optimization. In contrast with $L_p$-norm-based regularization, this regularizer is concerned with the spatial arrangement of weights within a weight matrix. This novel regularizer is an additive term for the loss function and is differentiable, simple and fast to compute, scale-invariant, requires a trivial amount of additional memory, and can easily be parallelized. Empirically this method yields approximately a one order-of-magnitude improvement in the number of nonzero model parameters required to achieve a given level of test accuracy when training LeNet300 on MNIST.