Towards Understanding Theoretical Advantages of Complex-Reaction Networks

TL;DR

This paper introduces complex-reaction networks, demonstrating their universal approximation capabilities and potential advantages over real-valued networks in terms of parameter efficiency and optimization landscape.

Contribution

It proves the universal approximation property for complex-reaction networks and shows they can approximate certain functions with polynomial parameters, unlike real networks requiring exponential parameters.

Findings

Complex-reaction networks can approximate radial functions with polynomial parameters.

The critical point set of complex-reaction networks is a subset of that of real networks.

Complex-reaction networks may facilitate easier optimization to find optimal solutions.

Abstract

Complex-valued neural networks have attracted increasing attention in recent years, while it remains open on the advantages of complex-valued neural networks in comparison with real-valued networks. This work takes one step on this direction by introducing the \emph{complex-reaction network} with fully-connected feed-forward architecture. We prove the universal approximation property for complex-reaction networks, and show that a class of radial functions can be approximated by a complex-reaction network using the polynomial number of parameters, whereas real-valued networks need at least exponential parameters to reach the same approximation level. For empirical risk minimization, our theoretical result shows that the critical point set of complex-reaction networks is a proper subset of that of real-valued networks, which may show some insights on finding the optimal solutions more easily for complex-reaction networks.