Discontinuous Piecewise Polynomial Neural Networks

TL;DR

This paper introduces discontinuous piecewise polynomial neural networks with localized activation functions, enhancing sparsity and computational efficiency, and demonstrates improved performance on MAGIC Gamma ray and MNIST datasets.

Contribution

It proposes a novel neural network architecture using polynomial activation functions with compact support, enabling sparse activations and improved performance over linear counterparts.

Findings

Higher order polynomials improve accuracy.

Sparse activations reduce computational load.

Effective on MAGIC Gamma and MNIST datasets.

Abstract

An artificial neural network is presented based on the idea of connections between units that are only active for a specific range of input values and zero outside that range (and so are not evaluated outside the active range). The connection function is represented by a polynomial with compact support. The finite range of activation allows for great activation sparsity in the network and means that theoretically you are able to add computational power to the network without increasing the computational time required to evaluate the network for a given input. The polynomial order ranges from first to fifth order. Unit dropout is used for regularization and a parameter free weight update is used. Better performance is obtained by moving from piecewise linear connections to piecewise quadratic, even better performance can be obtained by moving to higher order polynomials. The algorithm is tested on the MAGIC Gamma ray data set as well as the MNIST data set.