Learning activation functions from data using cubic spline interpolation

TL;DR

This paper introduces a data-dependent method for adapting activation functions in neural networks using cubic spline interpolation, allowing each neuron to learn its own shape and improve performance.

Contribution

It proposes a novel, efficient approach for neuron-specific, data-driven activation function adaptation using cubic spline interpolation with a damping criterion.

Findings

Improved performance on benchmark datasets.

Neuron-specific activation functions outperform fixed functions.

Method is computationally efficient and adaptable.

Abstract

Neural networks require a careful design in order to perform properly on a given task. In particular, selecting a good activation function (possibly in a data-dependent fashion) is a crucial step, which remains an open problem in the research community. Despite a large amount of investigations, most current implementations simply select one fixed function from a small set of candidates, which is not adapted during training, and is shared among all neurons throughout the different layers. However, neither two of these assumptions can be supposed optimal in practice. In this paper, we present a principled way to have data-dependent adaptation of the activation functions, which is performed independently for each neuron. This is achieved by leveraging over past and present advances on cubic spline interpolation, allowing for local adaptation of the functions around their regions of use. The resulting algorithm is relatively cheap to implement, and overfitting is counterbalanced by the inclusion of a novel damping criterion, which penalizes unwanted oscillations from a predefined shape. Experimental results validate the proposal over two well-known benchmarks.