Noisy Interpolation Learning with Shallow Univariate ReLU Networks

TL;DR

This paper rigorously analyzes how shallow univariate ReLU networks overfit noisy data, revealing tempered overfitting with respect to L1 loss but catastrophic overfitting with L2 loss.

Contribution

It provides the first rigorous analysis of overfitting in minimum norm regression with shallow ReLU networks, highlighting different behaviors under L1 and L2 losses.

Findings

Overfitting is tempered with respect to L1 loss.

Overfitting is catastrophic with respect to L2 loss.

Behavior differs when averaging over training sets.

Abstract

Understanding how overparameterized neural networks generalize despite perfect interpolation of noisy training data is a fundamental question. Mallinar et. al. 2022 noted that neural networks seem to often exhibit ``tempered overfitting'', wherein the population risk does not converge to the Bayes optimal error, but neither does it approach infinity, yielding non-trivial generalization. However, this has not been studied rigorously. We provide the first rigorous analysis of the overfitting behavior of regression with minimum norm ($\ell_2$ of weights), focusing on univariate two-layer ReLU networks. We show overfitting is tempered (with high probability) when measured with respect to the $L_1$ loss, but also show that the situation is more complex than suggested by Mallinar et. al., and overfitting is catastrophic with respect to the $L_2$ loss, or when taking an expectation over the training set.