A Statistical-Modelling Approach to Feedforward Neural Network Model Selection

TL;DR

This paper introduces a Bayesian information criterion-based method for selecting the structure of feedforward neural networks, improving model parsimony and true model recovery compared to traditional out-of-sample performance methods.

Contribution

It proposes a novel model selection approach for FNNs using BIC, enabling simultaneous input and hidden node selection with better statistical properties.

Findings

BIC-based selection improves true model recovery.

Method achieves parsimonious models with good out-of-sample performance.

Simulation and real data applications validate the approach.

Abstract

Feedforward neural networks (FNNs) can be viewed as non-linear regression models, where covariates enter the model through a combination of weighted summations and non-linear functions. Although these models have some similarities to the approaches used within statistical modelling, the majority of neural network research has been conducted outside of the field of statistics. This has resulted in a lack of statistically-based methodology, and, in particular, there has been little emphasis on model parsimony. Determining the input layer structure is analogous to variable selection, while the structure for the hidden layer relates to model complexity. In practice, neural network model selection is often carried out by comparing models using out-of-sample performance. However, in contrast, the construction of an associated likelihood function opens the door to information-criteria-based variable and architecture selection. A novel model selection method, which performs both input- and hidden-node selection, is proposed using the Bayesian information criterion (BIC) for FNNs. The choice of BIC over out-of-sample performance as the model selection objective function leads to an increased probability of recovering the true model, while parsimoniously achieving favourable out-of-sample performance. Simulation studies are used to evaluate and justify the proposed method, and applications on real data are investigated.