On improving deep learning generalization with adaptive sparse connectivity

TL;DR

This paper demonstrates that adaptive sparse neural networks with parameter-efficient training outperform fully-connected networks in generalization, using a novel training method combining sparse evolution and neuron pruning.

Contribution

It introduces a new training technique for sparse neural networks that improves generalization by combining Sparse Evolutionary Training with neuron pruning.

Findings

Sparse neural networks with adaptive connectivity outperform fully-connected networks in generalization.

The proposed method zeros out about 50% of neurons during training, maintaining competitive performance.

The approach uses a linear number of parameters relative to the number of neurons.

Abstract

Large neural networks are very successful in various tasks. However, with limited data, the generalization capabilities of deep neural networks are also very limited. In this paper, we empirically start showing that intrinsically sparse neural networks with adaptive sparse connectivity, which by design have a strict parameter budget during the training phase, have better generalization capabilities than their fully-connected counterparts. Besides this, we propose a new technique to train these sparse models by combining the Sparse Evolutionary Training (SET) procedure with neurons pruning. Operated on MultiLayer Perceptron (MLP) and tested on 15 datasets, our proposed technique zeros out around 50% of the hidden neurons during training, while having a linear number of parameters to optimize with respect to the number of neurons. The results show a competitive classification and generalization performance.