Understanding neural networks with reproducing kernel Banach spaces

TL;DR

This paper explores how reproducing kernel Banach spaces can characterize neural network function spaces, providing new theoretical insights into their properties and representations, especially for networks with ReLU activations.

Contribution

It proves a representer theorem for a broad class of reproducing kernel Banach spaces including infinite-width neural networks and characterizes their norms via inverse Radon transforms.

Findings

Proved a representer theorem for certain Banach spaces including neural networks.

Characterized the Banach space norm using inverse Radon transform for ReLU networks.

Extended recent theoretical results in neural network function space analysis.

Abstract

Characterizing the function spaces corresponding to neural networks can provide a way to understand their properties. In this paper we discuss how the theory of reproducing kernel Banach spaces can be used to tackle this challenge. In particular, we prove a representer theorem for a wide class of reproducing kernel Banach spaces that admit a suitable integral representation and include one hidden layer neural networks of possibly infinite width. Further, we show that, for a suitable class of ReLU activation functions, the norm in the corresponding reproducing kernel Banach space can be characterized in terms of the inverse Radon transform of a bounded real measure, with norm given by the total variation norm of the measure. Our analysis simplifies and extends recent results in [34,29,30].