Mean-field Analysis of Generalization Errors

TL;DR

This paper introduces a new framework using differential calculus on probability measures to analyze generalization errors, establishing conditions for an $ ext{O}(1/n)$ convergence rate in regularized empirical risk minimization, especially for neural networks.

Contribution

It develops a novel mathematical framework for understanding generalization errors via probability measure calculus and applies it to neural networks in the mean-field regime.

Findings

Generalization error converges at rate $ ext{O}(1/n)$ under certain conditions.

Framework applies to KL-regularized empirical risk minimization.

Conditions involve integrability and regularity of loss and activation functions.

Abstract

We propose a novel framework for exploring weak and $L_2$ generalization errors of algorithms through the lens of differential calculus on the space of probability measures. Specifically, we consider the KL-regularized empirical risk minimization problem and establish generic conditions under which the generalization error convergence rate, when training on a sample of size $n$, is $\mathcal{O}(1/n)$. In the context of supervised learning with a one-hidden layer neural network in the mean-field regime, these conditions are reflected in suitable integrability and regularity assumptions on the loss and activation functions.