Approximation in $L^p(\mu)$ with deep ReLU neural networks

TL;DR

This paper extends the theoretical understanding of ReLU neural networks' approximation capabilities from Lebesgue measure to general finite Borel measures, relevant for statistical learning.

Contribution

It generalizes existing approximation results for fixed-depth ReLU networks from Lebesgue measure to any finite Borel measure, including data distribution measures.

Findings

Generalization of approximation results to arbitrary finite Borel measures

Applicable to statistical learning scenarios with data distribution measures

Enhances theoretical understanding of ReLU networks' expressive power

Abstract

We discuss the expressive power of neural networks which use the non-smooth ReLU activation function $\varrho(x) = \max\{0,x\}$ by analyzing the approximation theoretic properties of such networks. The existing results mainly fall into two categories: approximation using ReLU networks with a fixed depth, or using ReLU networks whose depth increases with the approximation accuracy. After reviewing these findings, we show that the results concerning networks with fixed depth--- which up to now only consider approximation in $L^p(\lambda)$ for the Lebesgue measure $\lambda$--- can be generalized to approximation in $L^p(\mu)$, for any finite Borel measure $\mu$. In particular, the generalized results apply in the usual setting of statistical learning theory, where one is interested in approximation in $L^2(\mathbb{P})$, with the probability measure $\mathbb{P}$ describing the distribution of the data.