Newton's Method Backpropagation for Complex-Valued Holomorphic Multilayer Perceptrons

TL;DR

This paper develops Newton's method backpropagation algorithms for complex-valued holomorphic neural networks and demonstrates their efficiency improvements over traditional gradient descent methods through experimental comparisons.

Contribution

It introduces Newton's method backpropagation algorithms for complex holomorphic multilayer perceptrons and analyzes their convergence and performance.

Findings

Newton's method algorithms outperform gradient descent in training efficiency.

Polynomial activation functions improve convergence when used with Newton's method.

Experiments show significant reduction in training iterations with the proposed approach.

Abstract

The study of Newton's method in complex-valued neural networks faces many difficulties. In this paper, we derive Newton's method backpropagation algorithms for complex-valued holomorphic multilayer perceptrons, and investigate the convergence of the one-step Newton steplength algorithm for the minimization of real-valued complex functions via Newton's method. To provide experimental support for the use of holomorphic activation functions, we perform a comparison of using sigmoidal functions versus their Taylor polynomial approximations as activation functions by using the algorithms developed in this paper and the known gradient descent backpropagation algorithm. Our experiments indicate that the Newton's method based algorithms, combined with the use of polynomial activation functions, provide significant improvement in the number of training iterations required over the existing algorithms.