A New Constructive Method to Optimize Neural Network Architecture and Generalization

TL;DR

This paper introduces a constructive method that separates continuous and discontinuous parts of data to optimize neural network architecture, improve generalization, and reduce overfitting by fitting noise as singularities.

Contribution

It proposes a novel constructive approach that combines simple neural networks for continuous data with RBF networks for discontinuities, enhancing generalization and architecture optimization.

Findings

Effective separation of continuous and discontinuous data improves generalization.

Constructive approximation of jump discontinuities with RBF networks reduces overfitting.

The method maintains model simplicity while accurately fitting noisy data.

Abstract

In this paper, after analyzing the reasons of poor generalization and overfitting in neural networks, we consider some noise data as a singular value of a continuous function - jump discontinuity point. The continuous part can be approximated with the simplest neural networks, which have good generalization performance and optimal network architecture, by traditional algorithms such as constructive algorithm for feed-forward neural networks with incremental training, BP algorithm, ELM algorithm, various constructive algorithm, RBF approximation and SVM. At the same time, we will construct RBF neural networks to fit the singular value with every error in, and we prove that a function with jumping discontinuity points can be approximated by the simplest neural networks with a decay RBF neural networks in by each error, and a function with jumping discontinuity point can be constructively approximated by a decay RBF neural networks in by each error and the constructive part have no generalization influence to the whole machine learning system which will optimize neural network architecture and generalization performance, reduce the overfitting phenomenon by avoid fitting the noisy data.