Explicit integral representations and quantitative bounds for two-layer ReLU networks

TL;DR

This paper develops explicit integral representations for two-layer ReLU networks, providing bounds for approximation errors that depend on coefficients and data distribution, with connections to RKHS.

Contribution

It introduces new integral representations for ReLU networks and derives bounds that are independent of dimension and degree, enhancing understanding of their approximation capabilities.

Findings

Explicit integral representations for ReLU networks are constructed.

Quantitative bounds show errors depend on coefficients and data distribution, not dimension.

Connections to RKHS of the exponential kernel are established.

Abstract

An approach to construct explicit integral representations for two-layer ReLU networks is presented, which provides relatively simple representations for any multivariate polynomial. Quantitative bounds are provided for a particular, sharpened ReLU integral representation, which involves a harmonic extension and a projection. The bounds demonstrate that functions can be approximated with $L^{2}(\mathcal{D})$ errors that do not depend explicitly on dimension or degree, but rather the coefficients of their monomial expansions and the distribution $\mathcal{D}$. We also present a connection to the RKHS of the exponential kernel $K(x,y)=\exp\left(\left\langle x,y\right\rangle \right)$, and a very simple integral representation involving additionally multiplication via a fixed function which has better quantitative bounds.