Searching for Minimal Optimal Neural Networks

TL;DR

This paper develops a rigorous mathematical framework for selecting optimal neural network sizes, proving that the Adaptive group Lasso method can reliably identify the correct number of hidden nodes in one-hidden-layer networks.

Contribution

It provides the first theoretical guarantee for the destructive approach using Adaptive group Lasso in neural network size selection.

Findings

Adaptive group Lasso is consistent in reconstructing hidden nodes.

Theoretical guarantee established for the destructive neural network size reduction.

First asymptotic theory for the destructive technique in neural networks.

Abstract

Large neural network models have high predictive power but may suffer from overfitting if the training set is not large enough. Therefore, it is desirable to select an appropriate size for neural networks. The destructive approach, which starts with a large architecture and then reduces the size using a Lasso-type penalty, has been used extensively for this task. Despite its popularity, there is no theoretical guarantee for this technique. Based on the notion of minimal neural networks, we posit a rigorous mathematical framework for studying the asymptotic theory of the destructive technique. We prove that Adaptive group Lasso is consistent and can reconstruct the correct number of hidden nodes of one-hidden-layer feedforward networks with high probability. To the best of our knowledge, this is the first theoretical result establishing for the destructive technique.