Extrapolation and Spectral Bias of Neural Nets with Hadamard Product: a Polynomial Net Study

TL;DR

This paper analyzes polynomial neural networks with Hadamard products using NTK, revealing their superior extrapolation and spectral bias properties compared to standard neural networks, supported by theoretical and empirical evidence.

Contribution

It derives the finite-width NTK for polynomial neural networks with Hadamard products and demonstrates their advantages in extrapolation and spectral bias over standard neural networks.

Findings

PNNs can fit more complex functions in extrapolation regimes.

PNNs exhibit slower eigenvalue decay, leading to faster learning of high-frequency functions.

Empirical results confirm theoretical predictions across broader neural network classes.

Abstract

Neural tangent kernel (NTK) is a powerful tool to analyze training dynamics of neural networks and their generalization bounds. The study on NTK has been devoted to typical neural network architectures, but it is incomplete for neural networks with Hadamard products (NNs-Hp), e.g., StyleGAN and polynomial neural networks (PNNs). In this work, we derive the finite-width NTK formulation for a special class of NNs-Hp, i.e., polynomial neural networks. We prove their equivalence to the kernel regression predictor with the associated NTK, which expands the application scope of NTK. Based on our results, we elucidate the separation of PNNs over standard neural networks with respect to extrapolation and spectral bias. Our two key insights are that when compared to standard neural networks, PNNs can fit more complicated functions in the extrapolation regime and admit a slower eigenvalue decay of the respective NTK, leading to a faster learning towards high-frequency functions. Besides, our theoretical results can be extended to other types of NNs-Hp, which expand the scope of our work. Our empirical results validate the separations in broader classes of NNs-Hp, which provide a good justification for a deeper understanding of neural architectures.