Occam Factor for Gaussian Models With Unknown Variance Structure

TL;DR

This paper analyzes Bayesian model selection for Gaussian models with unknown variance structures, deriving exact asymptotics for Bayes factors and demonstrating superior performance over BIC through simulations.

Contribution

It provides a detailed theoretical framework for variance structure selection in Gaussian models, including exact asymptotics and a comparison with BIC.

Findings

Bayes factors correctly favor true models at known rates

Flexibility decomposes into BIC penalty plus an explicit term

Simulations show Bayesian evidence outperforms BIC in model selection

Abstract

We discuss model selection to determine whether the variance-covariance matrix of a multivariate Gaussian model with known mean should be considered to be a constant diagonal, a non-constant diagonal, or an arbitrary positive definite matrix. Of particular interest is the relationship between Bayesian evidence and the flexibility penalty due to Priebe and Rougier. For the case of an exponential family in canonical form equipped with a conjugate prior for the canonical parameter, flexibility may be exactly decomposed into the usual BIC likelihood penalty and a $O_p(1)$ term, the latter of which we explicitly compute. We also investigate the asymptotics of Bayes factors for linearly nested canonical exponential families equipped with conjugate priors; in particular, we find the exact rates at which Bayes factors correctly diverge in favor of the correct model: linearly and logarithmically in the number of observations when the full and nested models are true, respectively. Such theoretical considerations for the general case permit us to fully express the asymptotic behavior of flexibility and Bayes factors for the variance-covariance structure selection problem when we assume that the prior for the model precision is a member of the gamma/Wishart family of distributions or is uninformative. Simulations demonstrate evidence's immediate and superior performance in model selection compared to approximate criteria such as the BIC. We extend the framework to the multivariate Gaussian linear model with three data-driven examples.