Do ReLU Networks Have An Edge When Approximating Compactly-Supported Functions?

TL;DR

This paper demonstrates that ReLU neural networks with bilinear pooling can effectively approximate compactly-supported functions while capturing their support, establishing a new universality result in a refined topology.

Contribution

It introduces a refined topology on $L^1_{loc}$ to analyze structured approximation and proves the universality of ReLU networks with bilinear pooling for compactly-supported functions.

Findings

ReLU networks with bilinear pooling are universal in approximating compactly-supported functions.

Polynomial regressors and analytic networks are not universal in this space.

Quantitative bounds relate network architecture to function regularity and support properties.

Abstract

We study the problem of approximating compactly-supported integrable functions while implementing their support set using feedforward neural networks. Our first main result transcribes this "structured" approximation problem into a universality problem. We do this by constructing a refinement of the usual topology on the space $L^1_{\operatorname{loc}}(\mathbb{R}^d,\mathbb{R}^D)$ of locally-integrable functions in which compactly-supported functions can only be approximated in $L^1$-norm by functions with matching discretized support. We establish the universality of ReLU feedforward networks with bilinear pooling layers in this refined topology. Consequentially, we find that ReLU feedforward networks with bilinear pooling can approximate compactly supported functions while implementing their discretized support. We derive a quantitative uniform version of our universal approximation theorem on the dense subclass of compactly-supported Lipschitz functions. This quantitative result expresses the depth, width, and the number of bilinear pooling layers required to construct this ReLU network via the target function's regularity, the metric capacity and diameter of its essential support, and the dimensions of the inputs and output spaces. Conversely, we show that polynomial regressors and analytic feedforward networks are not universal in this space.