Efficient NTK using Dimensionality Reduction

TL;DR

This paper introduces a method to efficiently compute neural tangent kernels (NTK) for large neural networks by applying matrix factorization, significantly reducing computational costs while maintaining theoretical guarantees.

Contribution

The authors propose a matrix factorization approach to approximate NTK, enabling scalable analysis of large-width neural networks with reduced resource requirements.

Findings

Achieves similar theoretical guarantees as prior NTK analyses

Reduces training and inference costs through dimensionality reduction

Effective when input data dimension is comparable to the number of data points

Abstract

Recently, neural tangent kernel (NTK) has been used to explain the dynamics of learning parameters of neural networks, at the large width limit. Quantitative analyses of NTK give rise to network widths that are often impractical and incur high costs in time and energy in both training and deployment. Using a matrix factorization technique, we show how to obtain similar guarantees to those obtained by a prior analysis while reducing training and inference resource costs. The importance of our result further increases when the input points' data dimension is in the same order as the number of input points. More generally, our work suggests how to analyze large width networks in which dense linear layers are replaced with a low complexity factorization, thus reducing the heavy dependence on the large width.