Bayes factors and the geometry of discrete hierarchical loglinear models

TL;DR

This paper analyzes the behavior of Bayes factors in hierarchical loglinear models for discrete data, revealing how model sparsity and data support influence model comparison as hyperparameters tend to zero.

Contribution

It provides a mathematical characterization of Bayes factor asymptotics in hierarchical models, linking geometric properties of data support to model selection behavior.

Findings

Bayes factor behaves like α^{k_1 - k_2} as α tends to zero.

Sparser models are favored when data lies on lower-dimensional faces.

The geometric structure of data support influences Bayesian regularization.

Abstract

A standard tool for model selection in a Bayesian framework is the Bayes factor which compares the marginal likelihood of the data under two given different models. In this paper, we consider the class of hierarchical loglinear models for discrete data given under the form of a contingency table with multinomial sampling. We assume that the Diaconis-Ylvisaker conjugate prior is the prior distribution on the loglinear parameters and the uniform is the prior distribution on the space of models. Under these conditions, the Bayes factor between two models is a function of their prior and posterior normalizing constants. These constants are functions of the hyperparameters $(m,\alpha)$ which can be interpreted respectively as marginal counts and the total count of a fictive contingency table. We study the behaviour of the Bayes factor when $\alpha$ tends to zero. In this study two mathematical objects play a most important role. They are, first, the interior $C$ of the convex hull $\bar{C}$ of the support of the multinomial distribution for a given hierarchical loglinear model together with its faces and second, the characteristic function $\mathbb{J}_C$ of this convex set $C$. We show that, when $\alpha$ tends to 0, if the data lies on a face $F_i$ of $\bar{C_i},i=1,2$ of dimension $k_i$, the Bayes factor behaves like $\alpha^{k_1-k_2}$. This implies in particular that when the data is in $C_1$ and in $C_2$, i.e. when $k_i$ equals the dimension of model $J_i$, the sparser model is favored, thus confirming the idea of Bayesian regularization.