On the relationship between multivariate splines and infinitely-wide neural networks

TL;DR

This paper establishes a connection between multivariate splines and infinitely-wide neural networks, revealing a new random feature expansion that improves computational efficiency and numerical stability over traditional methods.

Contribution

It demonstrates that multivariate splines can be represented as infinitely-wide neural networks with a specific activation function, providing a novel and more effective random feature expansion.

Findings

The function space of the spline expansion is a Sobolev space with explicit derivative bounds.

The neural network-based random feature expansion outperforms Fourier features in theory and practice.

In one dimension, the neural network expansion has better leverage score scaling.

Abstract

We consider multivariate splines and show that they have a random feature expansion as infinitely wide neural networks with one-hidden layer and a homogeneous activation function which is the power of the rectified linear unit. We show that the associated function space is a Sobolev space on a Euclidean ball, with an explicit bound on the norms of derivatives. This link provides a new random feature expansion for multivariate splines that allow efficient algorithms. This random feature expansion is numerically better behaved than usual random Fourier features, both in theory and practice. In particular, in dimension one, we compare the associated leverage scores to compare the two random expansions and show a better scaling for the neural network expansion.