Lockout: Sparse Regularization of Neural Networks

TL;DR

This paper introduces a fast algorithm for obtaining all solutions to regularized neural network training problems with sparsity constraints, improving interpretability and often accuracy over dense models.

Contribution

It presents a novel, efficient method to compute all solutions for sparsity-inducing regularization in neural networks, extending solutions beyond linear models.

Findings

Sparse solutions often outperform dense models in accuracy.

Sparse neural networks are more interpretable.

Method makes neural networks competitive with state-of-the-art tabular data methods.

Abstract

Many regression and classification procedures fit a parameterized function $f(x;w)$ of predictor variables $x$ to data $\{x_{i},y_{i}\}_1^N$ based on some loss criterion $L(y,f)$. Often, regularization is applied to improve accuracy by placing a constraint $P(w)\leq t$ on the values of the parameters $w$. Although efficient methods exist for finding solutions to these constrained optimization problems for all values of $t\geq0$ in the special case when $f$ is a linear function, none are available when $f$ is non-linear (e.g. Neural Networks). Here we present a fast algorithm that provides all such solutions for any differentiable function $f$ and loss $L$, and any constraint $P$ that is an increasing monotone function of the absolute value of each parameter. Applications involving sparsity inducing regularization of arbitrary Neural Networks are discussed. Empirical results indicate that these sparse solutions are usually superior to their dense counterparts in both accuracy and interpretability. This improvement in accuracy can often make Neural Networks competitive with, and sometimes superior to, state-of-the-art methods in the analysis of tabular data.