Depth-induced NTK: Bridging Over-parameterized Neural Networks and Deep Neural Kernels

TL;DR

This paper introduces a depth-induced neural tangent kernel (NTK) that captures the role of network depth in over-parameterized neural networks, bridging the gap between deep architectures and kernel methods with theoretical analysis and experimental validation.

Contribution

We propose a novel depth-induced NTK based on shortcut architectures, extending NTK theory to finite-depth networks and analyzing its spectral and invariance properties.

Findings

The proposed kernel converges to a Gaussian process as depth increases.

It stabilizes kernel dynamics and reduces degeneration effects.

Experimental results confirm the theoretical advantages of the method.

Abstract

While deep learning has achieved remarkable success across a wide range of applications, its theoretical understanding of representation learning remains limited. Deep neural kernels provide a principled framework to interpret over-parameterized neural networks by mapping hierarchical feature transformations into kernel spaces, thereby combining the expressive power of deep architectures with the analytical tractability of kernel methods. Recent advances, particularly neural tangent kernels (NTKs) derived by gradient inner products, have established connections between infinitely wide neural networks and nonparametric Bayesian inference. However, the existing NTK paradigm has been predominantly confined to the infinite-width regime, while overlooking the representational role of network depth. To address this gap, we propose a depth-induced NTK kernel based on a shortcut-related architecture, which converges to a Gaussian process as the network depth approaches infinity. We theoretically analyze the training invariance and spectrum properties of the proposed kernel, which stabilizes the kernel dynamics and mitigates degeneration. Experimental results further underscore the effectiveness of our proposed method. Our findings significantly extend the existing landscape of the neural kernel theory and provide an in-depth understanding of deep learning and the scaling law.