Gibbs-Based Information Criteria and the Over-Parameterized Regime

TL;DR

This paper extends information criteria analysis to over-parameterized models using Gibbs algorithms, providing new BIC formulations that explain the double-descent phenomenon in high-dimensional settings.

Contribution

It introduces Gibbs-based AIC and BIC with information-theoretic penalties, extending classical criteria to over-parameterized models and analyzing their role in double-descent.

Findings

Gibbs-based BIC can effectively select high-dimensional models.

The analysis reveals a mismatch between marginal likelihood and population risk.

Experiments confirm the relevance of the new BIC in understanding double-descent.

Abstract

Double-descent refers to the unexpected drop in test loss of a learning algorithm beyond an interpolating threshold with over-parameterization, which is not predicted by information criteria in their classical forms due to the limitations in the standard asymptotic approach. We update these analyses using the information risk minimization framework and provide Akaike Information Criterion (AIC) and Bayesian Information Criterion (BIC) for models learned by the Gibbs algorithm. Notably, the penalty terms for the Gibbs-based AIC and BIC correspond to specific information measures, i.e., symmetrized KL information and KL divergence. We extend this information-theoretic analysis to over-parameterized models by providing two different Gibbs-based BICs to compute the marginal likelihood of random feature models in the regime where the number of parameters $p$ and the number of samples $n$ tend to infinity, with $p/n$ fixed. Our experiments demonstrate that the Gibbs-based BIC can select the high-dimensional model and reveal the mismatch between marginal likelihood and population risk in the over-parameterized regime, providing new insights to understand double-descent.