How Can We Be So Dense? The Benefits of Using Highly Sparse Representations

TL;DR

This paper demonstrates that highly sparse neural network representations, when combined with high dimensionality, offer improved robustness to noise and interference, while maintaining competitive accuracy and efficiency.

Contribution

It introduces the benefits of sparse representations in neural networks, analyzing their robustness and efficiency, supported by simulations on standard datasets.

Findings

Sparse representations are more robust to noise and interference.

High dimensionality enhances the effectiveness of sparse representations.

Sparse networks maintain accuracy while improving robustness and efficiency.

Abstract

Most artificial networks today rely on dense representations, whereas biological networks rely on sparse representations. In this paper we show how sparse representations can be more robust to noise and interference, as long as the underlying dimensionality is sufficiently high. A key intuition that we develop is that the ratio of the operable volume around a sparse vector divided by the volume of the representational space decreases exponentially with dimensionality. We then analyze computationally efficient sparse networks containing both sparse weights and activations. Simulations on MNIST and the Google Speech Command Dataset show that such networks demonstrate significantly improved robustness and stability compared to dense networks, while maintaining competitive accuracy. We discuss the potential benefits of sparsity on accuracy, noise robustness, hyperparameter tuning, learning speed, computational efficiency, and power requirements.