Uniform convergence may be unable to explain generalization in deep learning

TL;DR

This paper demonstrates that uniform convergence bounds often fail to explain why overparameterized deep networks generalize well, as these bounds can be vacuous or even increase with dataset size, highlighting limitations of current theoretical explanations.

Contribution

The paper provides empirical evidence and theoretical examples showing uniform convergence cannot fully account for generalization in deep learning, challenging existing bounds.

Findings

Uniform convergence bounds can increase with dataset size in practice.

Uniform convergence cannot explain generalization in certain overparameterized models.

Existing bounds often yield vacuous guarantees for models with low test error.

Abstract

Aimed at explaining the surprisingly good generalization behavior of overparameterized deep networks, recent works have developed a variety of generalization bounds for deep learning, all based on the fundamental learning-theoretic technique of uniform convergence. While it is well-known that many of these existing bounds are numerically large, through numerous experiments, we bring to light a more concerning aspect of these bounds: in practice, these bounds can {\em increase} with the training dataset size. Guided by our observations, we then present examples of overparameterized linear classifiers and neural networks trained by gradient descent (GD) where uniform convergence provably cannot "explain generalization" -- even if we take into account the implicit bias of GD {\em to the fullest extent possible}. More precisely, even if we consider only the set of classifiers output by GD, which have test errors less than some small $\epsilon$ in our settings, we show that applying (two-sided) uniform convergence on this set of classifiers will yield only a vacuous generalization guarantee larger than $1-\epsilon$. Through these findings, we cast doubt on the power of uniform convergence-based generalization bounds to provide a complete picture of why overparameterized deep networks generalize well.