The Interpolating Information Criterion for Overparameterized Models

TL;DR

This paper introduces the Interpolating Information Criterion, a new model selection measure for overparameterized models that accounts for prior misspecification and spectral properties, bridging classical and modern approaches.

Contribution

It establishes a duality between overparameterized and underparameterized models, enabling classical criteria to be adapted for modern overparameterized settings.

Findings

The criterion is numerically consistent with empirical results.

It accounts for prior misspecification and spectral properties.

Demonstrates duality between over- and underparameterized models.

Abstract

The problem of model selection is considered for the setting of interpolating estimators, where the number of model parameters exceeds the size of the dataset. Classical information criteria typically consider the large-data limit, penalizing model size. However, these criteria are not appropriate in modern settings where overparameterized models tend to perform well. For any overparameterized model, we show that there exists a dual underparameterized model that possesses the same marginal likelihood, thus establishing a form of Bayesian duality. This enables more classical methods to be used in the overparameterized setting, revealing the Interpolating Information Criterion, a measure of model quality that naturally incorporates the choice of prior into the model selection. Our new information criterion accounts for prior misspecification, geometric and spectral properties of the model, and is numerically consistent with known empirical and theoretical behavior in this regime.