Stable Minima of ReLU Neural Networks Suffer from the Curse of Dimensionality: The Neural Shattering Phenomenon

TL;DR

This paper demonstrates that flat minima in overparameterized ReLU neural networks lead to exponentially worse generalization in high-dimensional settings, due to a phenomenon called neural shattering, which explains why flat minima may fail in high dimensions.

Contribution

The paper provides the first systematic theoretical analysis showing how flat minima cause poor generalization in high-dimensional ReLU networks through neural shattering.

Findings

Flat solutions generalize poorly as input dimension increases.

Exponential deterioration of convergence rates for flat minima in high dimensions.

Neural shattering explains the failure of flat minima to generalize in high-dimensional spaces.

Abstract

We study the implicit bias of flatness / low (loss) curvature and its effects on generalization in two-layer overparameterized ReLU networks with multivariate inputs -- a problem well motivated by the minima stability and edge-of-stability phenomena in gradient-descent training. Existing work either requires interpolation or focuses only on univariate inputs. This paper presents new and somewhat surprising theoretical results for multivariate inputs. On two natural settings (1) generalization gap for flat solutions, and (2) mean-squared error (MSE) in nonparametric function estimation by stable minima, we prove upper and lower bounds, which establish that while flatness does imply generalization, the resulting rates of convergence necessarily deteriorate exponentially as the input dimension grows. This gives an exponential separation between the flat solutions compared to low-norm solutions (i.e., weight decay), which are known not to suffer from the curse of dimensionality. In particular, our minimax lower bound construction, based on a novel packing argument with boundary-localized ReLU neurons, reveals how flat solutions can exploit a kind of "neural shattering" where neurons rarely activate, but with high weight magnitudes. This leads to poor performance in high dimensions. We corroborate these theoretical findings with extensive numerical simulations. To the best of our knowledge, our analysis provides the first systematic explanation for why flat minima may fail to generalize in high dimensions.