Neural Spectrum Alignment: Empirical Study

TL;DR

This paper empirically investigates how the Neural Tangent Kernel (NTK) evolves during training, revealing that its top eigenfunctions align with the target function and significantly influence neural network learning and generalization.

Contribution

The study demonstrates that NTK properties are dynamic during training, with eigenfunctions aligning with the target, challenging the assumption of time-invariance in prior work.

Findings

NTK eigenfunctions align with target functions during training

Top eigenfunctions span the neural network output space

Learning rate decay affects kernel behavior and spectrum

Abstract

Expressiveness and generalization of deep models was recently addressed via the connection between neural networks (NNs) and kernel learning, where first-order dynamics of NN during a gradient-descent (GD) optimization were related to gradient similarity kernel, also known as Neural Tangent Kernel (NTK). In the majority of works this kernel is considered to be time-invariant, with its properties being defined entirely by NN architecture and independent of the learning task at hand. In contrast, in this paper we empirically explore these properties along the optimization and show that in practical applications the NTK changes in a very dramatic and meaningful way, with its top eigenfunctions aligning toward the target function learned by NN. Moreover, these top eigenfunctions serve as basis functions for NN output - a function represented by NN is spanned almost completely by them for the entire optimization process. Further, since the learning along top eigenfunctions is typically fast, their alignment with the target function improves the overall optimization performance. In addition, we study how the neural spectrum is affected by learning rate decay, typically done by practitioners, showing various trends in the kernel behavior. We argue that the presented phenomena may lead to a more complete theoretical understanding behind NN learning.