Separating Geometry from Probability in the Analysis of Generalization

TL;DR

This paper introduces a deterministic framework for analyzing machine learning generalization by examining the sensitivity of optimization solutions to data perturbations, bypassing traditional probabilistic assumptions.

Contribution

It proposes a novel variational approach that separates geometric and probabilistic aspects of generalization analysis, enabling deterministic bounds.

Findings

Provides generalization bounds based on data perturbation sensitivity.

Characterizes conditions under which out-of-sample performance closely matches in-sample results.

Offers a probabilistic interpretation of deterministic bounds using statistical assumptions.

Abstract

The goal of machine learning is to find models that minimize prediction error on data that has not yet been seen. Its operational paradigm assumes access to a dataset $S$ and articulates a scheme for evaluating how well a given model performs on an arbitrary sample. The sample can be $S$ (in which case we speak of ``in-sample'' performance) or some entirely new $S'$ (in which case we speak of ``out-of-sample'' performance). Traditional analysis of generalization assumes that both in- and out-of-sample data are i.i.d.\ draws from an infinite population. However, these probabilistic assumptions cannot be verified even in principle. This paper presents an alternative view of generalization through the lens of sensitivity analysis of solutions of optimization problems to perturbations in the problem data. Under this framework, generalization bounds are obtained by purely deterministic means and take the form of variational principles that relate in-sample and out-of-sample evaluations through an error term that quantifies how close out-of-sample data are to in-sample data. Statistical assumptions can then be used \textit{ex post} to characterize the situations when this error term is small (either on average or with high probability).