A Function-Space Stability Boundary for Generalization in Interpolating Learning Systems

TL;DR

This paper introduces a function-space stability boundary to understand when stability explains generalization in interpolating learning systems, revealing that stability is not always the sole factor.

Contribution

It models training as a function-space trajectory, proposes a stability certificate, and distinguishes regimes where stability explains generalization from those where it does not.

Findings

Stability certificates predict generalization across different optimizers and datasets.

Existence of interpolating regimes with small risk where stability does not hold.

Framework differentiates regimes where stability explains generalization from where it does not.

Abstract

Modern learning systems often interpolate training data while still generalizing well, yet it remains unclear when algorithmic stability explains this behavior. We model training as a function-space trajectory and measure sensitivity to single-sample perturbations along this trajectory. We propose a contractive propagation condition and a stability certificate obtained by unrolling the resulting recursion. A small certificate implies stability-based generalization, while we also prove that there exist interpolating regimes with small risk where such contractive sensitivity cannot hold, showing that stability is not a universal explanation. Experiments confirm that certificate growth predicts generalization differences across optimizers, step sizes, and dataset perturbations. The framework therefore identifies regimes where stability explains generalization and where alternative mechanisms must account for success.