Sparse Activity and Sparse Connectivity in Supervised Learning

TL;DR

This paper introduces a novel projection operator for enforcing sparsity in neural networks, enhancing classification performance on MNIST by promoting sparse activity and connectivity, and enabling end-to-end training.

Contribution

It develops a comprehensive theory for a sparseness-enforcing projection that is differentiable and can be integrated into neural networks for improved classification.

Findings

Sparse activity and connectivity improve MNIST classification.

The projection operator is differentiable and suitable for gradient-based training.

Combining both sparsity types yields the best performance.

Abstract

Sparseness is a useful regularizer for learning in a wide range of applications, in particular in neural networks. This paper proposes a model targeted at classification tasks, where sparse activity and sparse connectivity are used to enhance classification capabilities. The tool for achieving this is a sparseness-enforcing projection operator which finds the closest vector with a pre-defined sparseness for any given vector. In the theoretical part of this paper, a comprehensive theory for such a projection is developed. In conclusion, it is shown that the projection is differentiable almost everywhere and can thus be implemented as a smooth neuronal transfer function. The entire model can hence be tuned end-to-end using gradient-based methods. Experiments on the MNIST database of handwritten digits show that classification performance can be boosted by sparse activity or sparse connectivity. With a combination of both, performance can be significantly better compared to classical non-sparse approaches.