A Gap Between the Gaussian RKHS and Neural Networks: An Infinite-Center Asymptotic Analysis

TL;DR

This paper reveals a fundamental difference between Gaussian RKHS and neural network function spaces, showing that some functions well-represented by kernels are not captured by neural networks, especially on unbounded domains.

Contribution

It establishes that on unbounded domains, certain Gaussian RKHS functions have infinite neural network Banach space norm, highlighting a key gap between kernel methods and neural networks.

Findings

Gaussian RKHS functions can have infinite neural network Banach space norm on unbounded domains.

Neural networks and kernel methods differ significantly in the functions they can represent.

The study provides a theoretical foundation for understanding the limitations of neural networks compared to kernel methods.

Abstract

Recent works have characterized the function-space inductive bias of infinite-width bounded-norm single-hidden-layer neural networks as a kind of bounded-variation-type space. This novel neural network Banach space encompasses many classical multivariate function spaces, including certain Sobolev spaces and the spectral Barron spaces. Notably, this Banach space also includes functions that exhibit less classical regularity, such as those that only vary in a few directions. On bounded domains, it is well-established that the Gaussian reproducing kernel Hilbert space (RKHS) strictly embeds into this Banach space, demonstrating a clear gap between the Gaussian RKHS and the neural network Banach space. It turns out that when investigating these spaces on unbounded domains, e.g., all of $\mathbb{R}^d$, the story is fundamentally different. We establish the following fundamental result: Certain functions that lie in the Gaussian RKHS have infinite norm in the neural network Banach space. This provides a nontrivial gap between kernel methods and neural networks by exhibiting functions that kernel methods easily represent, whereas neural networks cannot.