Data-Driven Learning of Feedforward Neural Networks with Different Activation Functions

TL;DR

This paper extends a data-driven learning method for feedforward neural networks by incorporating various activation functions, demonstrating that sigmoid functions outperform others in approximating complex target functions.

Contribution

Introduces formulas for weights and biases for multiple activation functions within a data-driven FNN learning method, and evaluates their performance.

Findings

Sigmoid activation functions outperform others in complex function approximation.

The proposed formulas enable effective parameter setting for various activation functions.

Simulation results confirm the superiority of sigmoid functions in the studied context.

Abstract

This work contributes to the development of a new data-driven method (D-DM) of feedforward neural networks (FNNs) learning. This method was proposed recently as a way of improving randomized learning of FNNs by adjusting the network parameters to the target function fluctuations. The method employs logistic sigmoid activation functions for hidden nodes. In this study, we introduce other activation functions, such as bipolar sigmoid, sine function, saturating linear functions, reLU, and softplus. We derive formulas for their parameters, i.e. weights and biases. In the simulation study, we evaluate the performance of FNN data-driven learning with different activation functions. The results indicate that the sigmoid activation functions perform much better than others in the approximation of complex, fluctuated target functions.