Max-Margin Works while Large Margin Fails: Generalization without Uniform Convergence

TL;DR

This paper develops new theoretical generalization bounds for overparameterized models where traditional uniform convergence fails, highlighting the importance of near-max-margin classifiers in achieving good test performance.

Contribution

It introduces novel margin-based generalization bounds applicable in settings where uniform convergence does not hold, especially for near-max-margin classifiers.

Findings

Near-max-margin classifiers achieve low test loss above a certain signal-to-noise threshold.

Classical margin bounds and one-sided UC bounds can fail for near-max-margin classifiers.

Memorization can coexist with generalization due to the presence of generalizable components in the model.

Abstract

A major challenge in modern machine learning is theoretically understanding the generalization properties of overparameterized models. Many existing tools rely on uniform convergence (UC), a property that, when it holds, guarantees that the test loss will be close to the training loss, uniformly over a class of candidate models. Nagarajan and Kolter (2019) show that in certain simple linear and neural-network settings, any uniform convergence bound will be vacuous, leaving open the question of how to prove generalization in settings where UC fails. Our main contribution is proving novel generalization bounds in two such settings, one linear, and one non-linear. We study the linear classification setting of Nagarajan and Kolter, and a quadratic ground truth function learned via a two-layer neural network in the non-linear regime. We prove a new type of margin bound showing that above a certain signal-to-noise threshold, any near-max-margin classifier will achieve almost no test loss in these two settings. Our results show that near-max-margin is important: while any model that achieves at least a $(1 - \epsilon)$-fraction of the max-margin generalizes well, a classifier achieving half of the max-margin may fail terribly. Building on the impossibility results of Nagarajan and Kolter, under slightly stronger assumptions, we show that one-sided UC bounds and classical margin bounds will fail on near-max-margin classifiers. Our analysis provides insight on why memorization can coexist with generalization: we show that in this challenging regime where generalization occurs but UC fails, near-max-margin classifiers simultaneously contain some generalizable components and some overfitting components that memorize the data. The presence of the overfitting components is enough to preclude UC, but the near-extremal margin guarantees that sufficient generalizable components are present.