General Backpropagation Algorithm for Training Second-order Neural Networks

TL;DR

This paper introduces a general backpropagation algorithm for training neural networks composed of second-order neurons, which use quadratic operations to enhance nonlinear modeling capabilities.

Contribution

The paper develops a novel backpropagation algorithm specifically designed for second-order neural networks with quadratic neurons, expanding training methods for advanced neuron types.

Findings

The generalized BP algorithm successfully trains second-order neural networks.

Numerical studies confirm the effectiveness of the proposed training method.

Second-order neurons demonstrate strong nonlinear modeling abilities.

Abstract

The artificial neural network is a popular framework in machine learning. To empower individual neurons, we recently suggested that the current type of neurons could be upgraded to 2nd order counterparts, in which the linear operation between inputs to a neuron and the associated weights is replaced with a nonlinear quadratic operation. A single 2nd order neurons already has a strong nonlinear modeling ability, such as implementing basic fuzzy logic operations. In this paper, we develop a general backpropagation (BP) algorithm to train the network consisting of 2nd-order neurons. The numerical studies are performed to verify of the generalized BP algorithm.