Representation formulas and pointwise properties for Barron functions

TL;DR

This paper explores the mathematical structure of Barron space, characterizes its functions, and investigates the limitations and capabilities of infinitely wide two-layer neural networks with ReLU activation.

Contribution

It provides explicit representation formulas for Barron space, analyzes pointwise properties, and reveals new insights into the types of functions these networks can or cannot represent.

Findings

Functions with fractal or curved singular sets cannot be represented by Barron space.

The only $C^1$-diffeomorphisms in Barron space are affine.

Barron functions can be decomposed into bounded and positively one-homogeneous parts.

Abstract

We study the natural function space for infinitely wide two-layer neural networks with ReLU activation (Barron space) and establish different representation formulae. In two cases, we describe the space explicitly up to isomorphism. Using a convenient representation, we study the pointwise properties of two-layer networks and show that functions whose singular set is fractal or curved (for example distance functions from smooth submanifolds) cannot be represented by infinitely wide two-layer networks with finite path-norm. We use this structure theorem to show that the only $C^1$-diffeomorphisms which Barron space are affine. Furthermore, we show that every Barron function can be decomposed as the sum of a bounded and a positively one-homogeneous function and that there exist Barron functions which decay rapidly at infinity and are globally Lebesgue-integrable. This result suggests that two-layer neural networks may be able to approximate a greater variety of functions than commonly believed.