A Constructive Approach for Data-Driven Randomized Learning of Feedforward Neural Networks

TL;DR

This paper introduces an iterative, data-driven method for constructing feedforward neural networks by selectively adding hidden nodes based on their impact on training error, leading to faster convergence and more compact models.

Contribution

It extends existing data-driven random parameter generation methods by iteratively building the network architecture with adaptive thresholds for node acceptance.

Findings

Faster convergence compared to traditional methods

More compact neural network architectures

Effective in various application examples

Abstract

Feedforward neural networks with random hidden nodes suffer from a problem with the generation of random weights and biases as these are difficult to set optimally to obtain a good projection space. Typically, random parameters are drawn from an interval which is fixed before or adapted during the learning process. Due to the different functions of the weights and biases, selecting them both from the same interval is not a good idea. Recently more sophisticated methods of random parameters generation have been developed, such as the data-driven method proposed in \cite{Anon19}, where the sigmoids are placed in randomly selected regions of the input space and then their slopes are adjusted to the local fluctuations of the target function. In this work, we propose an extended version of this method, which constructs iteratively the network architecture. This method successively generates new hidden nodes and accepts them if the training error decreases significantly. The threshold of acceptance is adapted to the current training stage. At the beginning of the training process only those nodes which lead to the largest error reduction are accepted. Then, the threshold is reduced by half to accept those nodes which model the target function details more accurately. This leads to faster convergence and more compact network architecture, as it includes only "significant" neurons. Several application examples are given which confirm this thesis.