Learning the Number of Neurons in Deep Networks

TL;DR

This paper presents a method to automatically determine the optimal number of neurons in each layer of a deep network during training by leveraging structured sparsity, resulting in significantly more compact models without sacrificing accuracy.

Contribution

The paper introduces a novel approach using group sparsity regularization to automatically prune neurons during training, reducing network size by up to 80% while maintaining or improving performance.

Findings

Reduced network parameters by up to 80%

Maintained or improved accuracy after pruning

Automatically determines neuron count during learning

Abstract

Nowadays, the number of layers and of neurons in each layer of a deep network are typically set manually. While very deep and wide networks have proven effective in general, they come at a high memory and computation cost, thus making them impractical for constrained platforms. These networks, however, are known to have many redundant parameters, and could thus, in principle, be replaced by more compact architectures. In this paper, we introduce an approach to automatically determining the number of neurons in each layer of a deep network during learning. To this end, we propose to make use of structured sparsity during learning. More precisely, we use a group sparsity regularizer on the parameters of the network, where each group is defined to act on a single neuron. Starting from an overcomplete network, we show that our approach can reduce the number of parameters by up to 80\% while retaining or even improving the network accuracy.