A generalization gap estimation for overparameterized models via the Langevin functional variance

TL;DR

This paper introduces the Langevin functional variance as a practical tool to estimate the generalization gap in overparameterized models, providing an efficient approximation method suitable for neural networks and linear regression.

Contribution

It proposes the Langevin FV, an efficient approximation of the functional variance, enabling practical estimation of the generalization gap in overparameterized models.

Findings

Langevin FV accurately estimates the generalization gap.

The method is computationally efficient and gradient-based.

Validated on large overparameterized models.

Abstract

This paper discusses the estimation of the generalization gap, the difference between generalization performance and training performance, for overparameterized models including neural networks. We first show that a functional variance, a key concept in defining a widely-applicable information criterion, characterizes the generalization gap even in overparameterized settings where a conventional theory cannot be applied. As the computational cost of the functional variance is expensive for the overparameterized models, we propose an efficient approximation of the function variance, the Langevin approximation of the functional variance (Langevin FV). This method leverages only the $1$st-order gradient of the squared loss function, without referencing the $2$nd-order gradient; this ensures that the computation is efficient and the implementation is consistent with gradient-based optimization algorithms. We demonstrate the Langevin FV numerically by estimating the generalization gaps of overparameterized linear regression and non-linear neural network models, containing more than a thousand of parameters therein.