Pad\'e Neurons for Efficient Neural Models

TL;DR

This paper introduces Padé neurons (Paons), a novel non-linear neuron model inspired by Padé approximants, which enhances neural network efficiency and performance by providing diverse non-linearity and reducing layer count.

Contribution

The paper presents a new neuron model called Paons that generalizes existing models, offering stronger non-linearity and layer efficiency, and demonstrates its effectiveness in image tasks.

Findings

Paons outperform or match classic neurons in image tasks.

Paons achieve similar performance with fewer layers.

The implementation is open-source for community use.

Abstract

Neural networks commonly employ the McCulloch-Pitts neuron model, which is a linear model followed by a point-wise non-linear activation. Various researchers have already advanced inherently non-linear neuron models, such as quadratic neurons, generalized operational neurons, generative neurons, and super neurons, which offer stronger non-linearity compared to point-wise activation functions. In this paper, we introduce a novel and better non-linear neuron model called Pad\'e neurons (Paons), inspired by Pad\'e approximants. Paons offer several advantages, such as diversity of non-linearity, since each Paon learns a different non-linear function of its inputs, and layer efficiency, since Paons provide stronger non-linearity in much fewer layers compared to piecewise linear approximation. Furthermore, Paons include all previously proposed neuron models as special cases, thus any neuron model in any network can be replaced by Paons. We note that there has been a proposal to employ the Pad\'e approximation as a generalized point-wise activation function, which is fundamentally different from our model. To validate the efficacy of Paons, in our experiments, we replace classic neurons in some well-known neural image super-resolution, compression, and classification models based on the ResNet architecture with Paons. Our comprehensive experimental results and analyses demonstrate that neural models built by Paons provide better or equal performance than their classic counterparts with a smaller number of layers. The PyTorch implementation code for Paon is open-sourced at https://github.com/onur-keles/Paon.