Biologically Inspired Oscillating Activation Functions Can Bridge the Performance Gap between Biological and Artificial Neurons

TL;DR

This paper introduces four biologically inspired oscillating activation functions that enable single neurons to learn complex functions like XOR, improve training speed, and outperform existing activation functions on standard benchmarks.

Contribution

The paper proposes novel oscillating activation functions inspired by human pyramidal neurons, enhancing neural network capabilities and training efficiency.

Findings

Oscillating activation functions outperform popular functions on CIFAR datasets.

Single neurons with these functions can learn XOR, a complex logical operation.

Networks trained with these functions converge faster and require fewer layers.

Abstract

The recent discovery of special human neocortical pyramidal neurons that can individually learn the XOR function highlights the significant performance gap between biological and artificial neurons. The output of these pyramidal neurons first increases to a maximum with input and then decreases. Artificial neurons with similar characteristics can be designed with oscillating activation functions. Oscillating activation functions have multiple zeros allowing single neurons to have multiple hyper-planes in their decision boundary. This enables even single neurons to learn the XOR function. This paper proposes four new oscillating activation functions inspired by human pyramidal neurons that can also individually learn the XOR function. Oscillating activation functions are non-saturating for all inputs unlike popular activation functions, leading to improved gradient flow and faster convergence. Using oscillating activation functions instead of popular monotonic or non-monotonic single-zero activation functions enables neural networks to train faster and solve classification problems with fewer layers. An extensive comparison of 23 activation functions on CIFAR 10, CIFAR 100, and Imagentte benchmarks is presented and the oscillating activation functions proposed in this paper are shown to outperform all known popular activation functions.