On Uniform Convergence and Low-Norm Interpolation Learning

TL;DR

This paper investigates the limitations of uniform convergence in explaining the success of minimum-norm interpolators in noisy linear regression, proposing a weaker notion of uniform convergence to better understand their generalization.

Contribution

It introduces a weaker uniform convergence concept that can explain the generalization of low-norm interpolators, addressing limitations of traditional uniform convergence.

Findings

Uniform convergence bounds cannot explain learning in norm balls.

A weaker uniform convergence notion can account for the success of minimum-norm interpolators.

The approach bounds the generalization error of low-norm interpolating predictors.

Abstract

We consider an underdetermined noisy linear regression model where the minimum-norm interpolating predictor is known to be consistent, and ask: can uniform convergence in a norm ball, or at least (following Nagarajan and Kolter) the subset of a norm ball that the algorithm selects on a typical input set, explain this success? We show that uniformly bounding the difference between empirical and population errors cannot show any learning in the norm ball, and cannot show consistency for any set, even one depending on the exact algorithm and distribution. But we argue we can explain the consistency of the minimal-norm interpolator with a slightly weaker, yet standard, notion: uniform convergence of zero-error predictors in a norm ball. We use this to bound the generalization error of low- (but not minimal-) norm interpolating predictors.