Spectral Complexity-scaled Generalization Bound of Complex-valued Neural Networks

TL;DR

This paper establishes a novel generalization bound for complex-valued neural networks based on spectral complexity, supported by theoretical derivations and empirical experiments across multiple datasets, highlighting the spectral norm product's role in generalization.

Contribution

It is the first work to derive a spectral complexity-based generalization bound specifically for complex-valued neural networks, including sequential data scenarios.

Findings

Spectral complexity correlates significantly with generalization ability.

Theoretical bounds are derived using Maurey Sparsification and Dudley Entropy.

Empirical results across datasets support the spectral norm product as a key factor.

Abstract

Complex-valued neural networks (CVNNs) have been widely applied to various fields, especially signal processing and image recognition. However, few works focus on the generalization of CVNNs, albeit it is vital to ensure the performance of CVNNs on unseen data. This paper is the first work that proves a generalization bound for the complex-valued neural network. The bound scales with the spectral complexity, the dominant factor of which is the spectral norm product of weight matrices. Further, our work provides a generalization bound for CVNNs when training data is sequential, which is also affected by the spectral complexity. Theoretically, these bounds are derived via Maurey Sparsification Lemma and Dudley Entropy Integral. Empirically, we conduct experiments by training complex-valued convolutional neural networks on different datasets: MNIST, FashionMNIST, CIFAR-10, CIFAR-100, Tiny ImageNet, and IMDB. Spearman's rank-order correlation coefficients and the corresponding p values on these datasets give strong proof that the spectral complexity of the network, measured by the weight matrices spectral norm product, has a statistically significant correlation with the generalization ability.