Robust model selection using likelihood as data

TL;DR

This paper proposes a robust model selection method that models likelihood values directly, using a multivariate normal approach to estimate divergences and quantify uncertainty, improving robustness under model misspecification.

Contribution

It introduces a novel likelihood-based approach for model selection that accounts for uncertainty and is robust to misspecification, with theoretical guarantees.

Findings

Method provides calibrated uncertainty estimates.

Approach is computationally simple and interpretable.

Theoretical guarantees including consistency are established.

Abstract

Model selection is a central task in statistics, but standard methods are not robust in misspecified settings where the true data-generating process (DGP) is not in the set of candidate models. The key limitation is that existing methods -- including information criteria and Bayesian posteriors -- do not quantify uncertainty about how well each candidate model approximates the true DGP. In this paper, we introduce a novel approach to model selection based on modeling the likelihood values themselves. Specifically, given $K$ candidate models and $n$ observations, we view the $n\times K$ matrix of negative log-likelihood values as a random data matrix and observe that the expectation of each row is equal to the vector of Kullback--Leibler divergences between the $K$ models and the true DGP, up to an additive constant. We use a multivariate normal model to estimate and quantify uncertainty in this expectation, providing calibrated inferences for robust model selection under misspecification. The procedure is easy to compute, interpretable, and comes with theoretical guarantees, including consistency.