Limiting Network Size within Finite Bounds for Optimization

TL;DR

This paper provides a theoretical framework for bounding the size of neural network layers, ensuring optimal classification performance with minimal complexity, supported by experimental validation.

Contribution

It introduces a theoretical justification for the necessary size of hidden layers in shallow networks for binary classification, establishing bounds on network width.

Findings

Bounded hidden layer size leads to efficient training.

Theoretical bounds are validated experimentally on multiple datasets.

Optimal network size improves classification with minimal overfitting.

Abstract

Largest theoretical contribution to Neural Networks comes from VC Dimension which characterizes the sample complexity of classification model in a probabilistic view and are widely used to study the generalization error. So far in the literature the VC Dimension has only been used to approximate the generalization error bounds on different Neural Network architectures. VC Dimension has not yet been implicitly or explicitly stated to fix the network size which is important as the wrong configuration could lead to high computation effort in training and leads to over fitting. So there is a need to bound these units so that task can be computed with only sufficient number of parameters. For binary classification tasks shallow networks are used as they have universal approximation property and it is enough to size the hidden layer width for such networks. The paper brings out a theoretical justification on required attribute size and its corresponding hidden layer dimension for a given sample set that gives an optimal binary classification results with minimum training complexity in a single layered feed forward network framework. The paper also establishes proof on the existence of bounds on the width of the hidden layer and its range subjected to certain conditions. Findings in this paper are experimentally analyzed on three different dataset using Mathlab 2018 (b) software.