On Bayesian Exponentially Embedded Family for Model Order Selection

TL;DR

This paper introduces a Bayesian model order selection method using the exponentially embedded family, which can incorporate vague or improper priors and relates to information theory and frequentist approaches.

Contribution

It develops a Bayesian EEF rule that objectively uses vague priors, links penalty terms to mutual information, and unifies Bayesian and frequentist model selection strategies.

Findings

The penalty term combines parameter dimension and mutual information.

The Bayesian EEF with Jeffreys prior matches frequentist EEF rules.

Illustrated with linear model order selection examples.

Abstract

In this paper, we derive a Bayesian model order selection rule by using the exponentially embedded family method, termed Bayesian EEF. Unlike many other Bayesian model selection methods, the Bayesian EEF can use vague proper priors and improper noninformative priors to be objective in the elicitation of parameter priors. Moreover, the penalty term of the rule is shown to be the sum of half of the parameter dimension and the estimated mutual information between parameter and observed data. This helps to reveal the EEF mechanism in selecting model orders and may provide new insights into the open problems of choosing an optimal penalty term for model order selection and choosing a good prior from information theoretic viewpoints. The important example of linear model order selection is given to illustrate the algorithms and arguments. Lastly, the Bayesian EEF that uses Jeffreys prior coincides with the EEF rule derived by frequentist strategies. This shows another interesting relationship between the frequentist and Bayesian philosophies for model selection.