Neural network with optimal neuron activation functions based on additive Gaussian process regression

TL;DR

This paper introduces a novel method using additive Gaussian process regression to create optimal, individual neuron activation functions, enhancing neural network efficiency and accuracy without complex non-linear optimization.

Contribution

It presents a new approach to design neuron activation functions tailored to each neuron using additive GPR, avoiding non-linear fitting and improving neural network performance.

Findings

Outperforms conventional neural networks in high-accuracy regimes

Reduces overfitting compared to traditional methods

Eliminates the need for non-linear optimization in training

Abstract

Feed-forward neural networks (NN) are a staple machine learning method widely used in many areas of science and technology. While even a single-hidden layer NN is a universal approximator, its expressive power is limited by the use of simple neuron activation functions (such as sigmoid functions) that are typically the same for all neurons. More flexible neuron activation functions would allow using fewer neurons and layers and thereby save computational cost and improve expressive power. We show that additive Gaussian process regression (GPR) can be used to construct optimal neuron activation functions that are individual to each neuron. An approach is also introduced that avoids non-linear fitting of neural network parameters. The resulting method combines the advantage of robustness of a linear regression with the higher expressive power of a NN. We demonstrate the approach by fitting the potential energy surfaces of the water molecule and formaldehyde. Without requiring any non-linear optimization, the additive GPR based approach outperforms a conventional NN in the high accuracy regime, where a conventional NN suffers more from overfitting.