Banach Space Representer Theorems for Neural Networks and Ridge Splines

TL;DR

This paper introduces a variational framework linking neural networks to inverse problems and spline theory, revealing how regularizers influence network solutions and their generalization.

Contribution

It establishes a representer theorem for neural networks in a Banach space setting, connecting them to polynomial ridge splines and providing insights into regularization effects.

Findings

Neural networks can be characterized as solutions to inverse problems with TV-like regularization.

Regularizers like weight decay relate to the properties of the learned functions.

The framework offers insights into the generalization capabilities of neural networks.

Abstract

We develop a variational framework to understand the properties of the functions learned by neural networks fit to data. We propose and study a family of continuous-domain linear inverse problems with total variation-like regularization in the Radon domain subject to data fitting constraints. We derive a representer theorem showing that finite-width, single-hidden layer neural networks are solutions to these inverse problems. We draw on many techniques from variational spline theory and so we propose the notion of polynomial ridge splines, which correspond to single-hidden layer neural networks with truncated power functions as the activation function. The representer theorem is reminiscent of the classical reproducing kernel Hilbert space representer theorem, but we show that the neural network problem is posed over a non-Hilbertian Banach space. While the learning problems are posed in the continuous-domain, similar to kernel methods, the problems can be recast as finite-dimensional neural network training problems. These neural network training problems have regularizers which are related to the well-known weight decay and path-norm regularizers. Thus, our result gives insight into functional characteristics of trained neural networks and also into the design neural network regularizers. We also show that these regularizers promote neural network solutions with desirable generalization properties.