Optimal bump functions for shallow ReLU networks: Weight decay, depth separation and the curse of dimensionality

TL;DR

This paper investigates how shallow ReLU networks interpolate radially symmetric data with weight decay regularization, revealing exponential growth in regularizer with dimension and demonstrating that deeper networks can avoid the curse of dimensionality.

Contribution

It provides a theoretical analysis of optimal bump functions in shallow ReLU networks, highlighting the impact of depth on avoiding the curse of dimensionality.

Findings

Weight decay regularizer grows exponentially with dimension in shallow networks.

Deeper networks can approximate target functions without the curse of dimensionality.

Unique radially symmetric minimizer exists with growth rates depending on dimension.

Abstract

In this note, we study how neural networks with a single hidden layer and ReLU activation interpolate data drawn from a radially symmetric distribution with target labels 1 at the origin and 0 outside the unit ball, if no labels are known inside the unit ball. With weight decay regularization and in the infinite neuron, infinite data limit, we prove that a unique radially symmetric minimizer exists, whose weight decay regularizer and Lipschitz constant grow as $d$ and $\sqrt{d}$ respectively. We furthermore show that the weight decay regularizer grows exponentially in $d$ if the label $1$ is imposed on a ball of radius $\varepsilon$ rather than just at the origin. By comparison, a neural networks with two hidden layers can approximate the target function without encountering the curse of dimensionality.