A Special Case of Quadratic Extrapolation Under the Neural Tangent Kernel

TL;DR

This paper investigates the unique case of quadratic extrapolation near the origin under the neural tangent kernel, revealing insights into the behavior of ReLU MLPs in out-of-distribution scenarios.

Contribution

It introduces the analysis of quadratic extrapolation at the origin under NTK, a previously underexplored special case, highlighting the differences from far-out extrapolation.

Findings

Quadratic extrapolation occurs near the origin under NTK.

The feature map's properties influence extrapolation behavior.

Distinct behaviors are observed between near-origin and far-out extrapolation.

Abstract

It has been demonstrated both theoretically and empirically that the ReLU MLP tends to extrapolate linearly for an out-of-distribution evaluation point. The machine learning literature provides ample analysis with respect to the mechanisms to which linearity is induced. However, the analysis of extrapolation at the origin under the NTK regime remains a more unexplored special case. In particular, the infinite-dimensional feature map induced by the neural tangent kernel is not translationally invariant. This means that the study of an out-of-distribution evaluation point very far from the origin is not equivalent to the evaluation of a point very near the origin. And since the feature map is rotation invariant, these two special cases may represent the most canonically extreme bounds of ReLU NTK extrapolation. Ultimately, it is this loose recognition of the two special cases of extrapolation that motivate the discovery of quadratic extrapolation for an evaluation close to the origin.