Pointwise Generalization in Deep Neural Networks

TL;DR

This paper introduces a pointwise generalization theory for deep neural networks, providing a new statistical framework that explains their generalization behavior through a hypothesis-dependent, feature-aware complexity measure.

Contribution

It develops a novel pointwise Riemannian Dimension to characterize learned features, leading to tighter generalization bounds and deeper understanding of deep network tractability.

Findings

Pointwise Riemannian Dimension decreases with over-parameterization.

Feature compression is substantial and observable.

Generalization bounds are significantly tighter than previous approaches.

Abstract

We address the fundamental question of why deep neural networks generalize by establishing a pointwise generalization theory for fully connected networks. This framework resolves long-standing barriers to characterizing the rich nonlinear feature-learning regime and builds a new statistical foundation for representation learning. For each trained model, we characterize the hypothesis via a pointwise Riemannian Dimension, derived from the eigenvalues of the learned feature representations across layers. This establishes a principled framework for deriving hypothesis-dependent, representation-aware generalization bounds. These bounds offer a systematic upgrade over approaches based on model size, products of norms, and infinite-width linearizations, yielding guarantees that are orders of magnitude tighter in both theory and experiment. Analytically, we identify the structural properties and mathematical principles that explain the tractability of deep networks. Empirically, the pointwise Riemannian Dimension exhibits substantial feature compression, decreases with increased over-parameterization, and captures the implicit bias of optimizers. Taken together, our results indicate that deep networks are mathematically tractable in practical regimes and that their generalization is sharply explained by pointwise, feature-spectrum-aware complexity.