A New Type of Neurons for Machine Learning

TL;DR

This paper proposes replacing the traditional linear inner product in neural network neurons with a quadratic function, creating second-order neurons that enhance individual neuron capabilities and potentially improve network optimization.

Contribution

Introduces second-order neurons with quadratic input processing, offering a novel approach to upgrade neural network architecture and optimization methods.

Findings

Numerical examples demonstrate feasibility of second-order neurons.

Second-order neurons show potential for improved network performance.

Discussion of future research directions.

Abstract

In machine learning, the use of an artificial neural network is the mainstream approach. Such a network consists of layers of neurons. These neurons are of the same type characterized by the two features: (1) an inner product of an input vector and a matching weighting vector of trainable parameters and (2) a nonlinear excitation function. Here we investigate the possibility of replacing the inner product with a quadratic function of the input vector, thereby upgrading the 1st order neuron to the 2nd order neuron, empowering individual neurons, and facilitating the optimization of neural networks. Also, numerical examples are provided to illustrate the feasibility and merits of the 2nd order neurons. Finally, further topics are discussed.