Mixed Newton Method for Optimization in Complex Spaces

TL;DR

This paper extends the Mixed Newton Method from complex to real spaces, demonstrating its convergence properties and effectiveness in training neural networks with real and complex parameters.

Contribution

It introduces a regularization approach that preserves convergence and prevents complex minima, adapting the method for real-valued optimization.

Findings

Regularization maintains local convergence properties.

Method effectively trains neural networks with complex parameters.

Prevents convergence to undesired complex minima.

Abstract

In this paper, we modify and apply the recently introduced Mixed Newton Method, which is originally designed for minimizing real-valued functions of complex variables, to the minimization of real-valued functions of real variables by extending the functions to complex space. We show that arbitrary regularizations preserve the favorable local convergence properties of the method, and construct a special type of regularization used to prevent convergence to complex minima. We compare several variants of the method applied to training neural networks with real and complex parameters.