The Recurrent Neural Tangent Kernel
Sina Alemohammad, Zichao Wang, Randall Balestriero, Richard Baraniuk

TL;DR
This paper introduces the Recurrent Neural Tangent Kernel (RNTK), extending the NTK framework to RNNs, providing new theoretical insights and demonstrating superior performance on various datasets.
Contribution
The paper develops the RNTK, enabling analysis of overparametrized RNNs and their ability to handle inputs of varying lengths, with empirical validation showing performance improvements.
Findings
RNTK effectively compares inputs of different lengths.
RNTK outperforms standard NTKs on multiple datasets.
Theoretical characterization of RNTK weights and behavior.
Abstract
The study of deep neural networks (DNNs) in the infinite-width limit, via the so-called neural tangent kernel (NTK) approach, has provided new insights into the dynamics of learning, generalization, and the impact of initialization. One key DNN architecture remains to be kernelized, namely, the recurrent neural network (RNN). In this paper we introduce and study the Recurrent Neural Tangent Kernel (RNTK), which provides new insights into the behavior of overparametrized RNNs. A key property of the RNTK should greatly benefit practitioners is its ability to compare inputs of different length. To this end, we characterize how the RNTK weights different time steps to form its output under different initialization parameters and nonlinearity choices. A synthetic and 56 real-world data experiments demonstrate that the RNTK offers significant performance gains over other kernels, including…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
Taxonomy
TopicsNeural Networks and Applications · Gaussian Processes and Bayesian Inference · Domain Adaptation and Few-Shot Learning
