Analyzing the Neural Tangent Kernel of Periodically Activated Coordinate Networks

TL;DR

This paper provides a theoretical analysis of periodically activated neural networks through their Neural Tangent Kernel, showing they are more well-behaved than ReLU networks and exploring their memorization capacity.

Contribution

It offers the first theoretical bounds on the NTK of periodically activated networks and compares their properties to ReLU networks, supported by empirical verification.

Findings

Periodically activated networks have better NTK properties than ReLU networks.

Theoretical bounds on the minimum eigenvalue of the NTK are derived.

Empirical results confirm the theoretical predictions.

Abstract

Recently, neural networks utilizing periodic activation functions have been proven to demonstrate superior performance in vision tasks compared to traditional ReLU-activated networks. However, there is still a limited understanding of the underlying reasons for this improved performance. In this paper, we aim to address this gap by providing a theoretical understanding of periodically activated networks through an analysis of their Neural Tangent Kernel (NTK). We derive bounds on the minimum eigenvalue of their NTK in the finite width setting, using a fairly general network architecture which requires only one wide layer that grows at least linearly with the number of data samples. Our findings indicate that periodically activated networks are \textit{notably more well-behaved}, from the NTK perspective, than ReLU activated networks. Additionally, we give an application to the memorization capacity of such networks and verify our theoretical predictions empirically. Our study offers a deeper understanding of the properties of periodically activated neural networks and their potential in the field of deep learning.