The Positivity of the Neural Tangent Kernel

TL;DR

This paper proves that the Neural Tangent Kernel (NTK) is strictly positive definite for any non-polynomial activation function in feedforward neural networks of any depth, enhancing understanding of network memorization capacity.

Contribution

It provides a sharp, general result that the NTK is strictly positive definite for all non-polynomial activations, improving previous bounds and characterizations.

Findings

NTK is strictly positive definite for non-polynomial activations

Results apply to feedforward networks of any depth

Introduces a novel characterization of polynomial functions

Abstract

The Neural Tangent Kernel (NTK) has emerged as a fundamental concept in the study of wide Neural Networks. In particular, it is known that the positivity of the NTK is directly related to the memorization capacity of sufficiently wide networks, i.e., to the possibility of reaching zero loss in training, via gradient descent. Here we will improve on previous works and obtain a sharp result concerning the positivity of the NTK of feedforward networks of any depth. More precisely, we will show that, for any non-polynomial activation function, the NTK is strictly positive definite. Our results are based on a novel characterization of polynomial functions which is of independent interest.