Complex-valued Neural Networks -- Theory and Analysis

TL;DR

This paper provides a comprehensive overview of complex-valued neural networks, covering their structures, theory, learning algorithms, modules, and future research directions, highlighting recent advances and practical implementations.

Contribution

It offers a detailed synthesis of CVNN theory, including activation functions, optimization, modules, and software tools, advancing understanding of their dynamics and recent developments.

Findings

Analysis of complex activation functions and their differentiability

Discussion of complex backpropagation using Wirtinger calculus

Overview of modules like complex batch normalization and initialization

Abstract

Complex-valued neural networks (CVNNs) have recently been successful in various pioneering areas which involve wave-typed information and frequency-domain processing. This work addresses different structures and classification of CVNNs. The theory behind complex activation functions, implications related to complex differentiability and special activations for CVNN output layers are presented. The work also discusses CVNN learning and optimization using gradient and non-gradient based algorithms. Complex Backpropagation utilizing complex chain rule is also explained in terms of Wirtinger calculus. Moreover, special modules for building CVNN models, such as complex batch normalization and complex random initialization are also discussed. The work also highlights libraries and software blocks proposed for CVNN implementations and discusses future directions. The objective of this work is to understand the dynamics and most recent developments of CVNNs.