Improving neural networks with bunches of neurons modeled by Kumaraswamy units: Preliminary study

TL;DR

This paper introduces Kumaraswamy units, a new neural activation function modeled by the Kumaraswamy distribution, demonstrating improved performance over traditional activations on MNIST.

Contribution

It proposes a novel activation function called Kumaraswamy unit, inspired by the Kumaraswamy distribution, and evaluates its effectiveness in neural networks.

Findings

Significant reduction in test classification error with Kumaraswamy units

Lower test cross-entropy compared to ReLU and sigmoid

Effective in shallow neural network on MNIST

Abstract

Deep neural networks have recently achieved state-of-the-art results in many machine learning problems, e.g., speech recognition or object recognition. Hitherto, work on rectified linear units (ReLU) provides empirical and theoretical evidence on performance increase of neural networks comparing to typically used sigmoid activation function. In this paper, we investigate a new manner of improving neural networks by introducing a bunch of copies of the same neuron modeled by the generalized Kumaraswamy distribution. As a result, we propose novel non-linear activation function which we refer to as Kumaraswamy unit which is closely related to ReLU. In the experimental study with MNIST image corpora we evaluate the Kumaraswamy unit applied to single-layer (shallow) neural network and report a significant drop in test classification error and test cross-entropy in comparison to sigmoid unit, ReLU and Noisy ReLU.