Bases of Lebesgue spaces formed by neural networks

TL;DR

This paper studies the properties of neural network-based bases in Lebesgue spaces, extending previous univariate results to multivariate settings and analyzing their basis properties in various L_q spaces.

Contribution

It introduces a new approach to analyze neural network-based bases in Lebesgue spaces, generalizing univariate systems to multivariate cases with dimension-independent properties.

Findings

Univariate system is a Schauder basis in L_q(0,1) for 1<q<∞.

Tensor products form Schauder bases in L_q(0,1)^n for all n≥2 and 1<q<∞.

Inner product-based systems extend to arbitrary dimensions n∈ℕ, unlike tensor product systems.

Abstract

The seminal work of Daubechies, DeVore, Foucart, Hanin, and Petrova introduced in 2022 a sequence of univariate piece-wise linear functions, which resemble the classical Fourier basis and which, at the same time, can be easily reproduced by artificial neural networks with ReLU activation function. We give an alternative way how to calculate the inner products of functions from this system and discuss the spectral properties of the Gram matrix generated by this system. The univariate system was later generalized to the multivariate setting by two of the authors of this work. Instead of the usual tensor product construction, this generalization relied on the inner products inside of the argument of the univariate sequence. It turned out that such a system forms a Riesz basis of $L_2(0,1)^n$ for every $n\ge 1$ with Riesz constants independent of $n$. In this work, we investigate the properties of these new sequences of functions in $L_q(0,1)^n$ for $q\not =2.$ First, we show that the univariate system is a Schauder basis in $L_q(0,1)$ for every $1<q<\infty$. By a general argument, it follows that the tensor products of this system also form a Schauder basis in $L_q(0,1)^n$ for every $n\ge 2$ and $1<q<\infty.$ The same fact can also be shown by measuring the distance of the tensor product system to the classical multivariate Fourier basis, but - surprisingly - this argument only works for $n\le 3$. If, on the other hand, we replace the outer tensor products by inner products directly in the argument of the univariate system, the same approach is applicable for an arbitrary dimension $n\in{\mathbb N}.$