Three Approaches to Probability Model Selection

TL;DR

This paper compares three methods for selecting probability models, highlighting their relationships and applicability, especially in computationally efficient and AI contexts, with a focus on penalized likelihood frameworks.

Contribution

It unifies three model selection approaches under a penalized likelihood framework and demonstrates their relationships and practical advantages in AI applications.

Findings

All three methods can be viewed as maximizing a penalized likelihood.

The second approach reduces to the first with suitable priors.

The third approach reduces to the second for a specific effectiveness ratio.

Abstract

This paper compares three approaches to the problem of selecting among probability models to fit data (1) use of statistical criteria such as Akaike's information criterion and Schwarz's "Bayesian information criterion," (2) maximization of the posterior probability of the model, and (3) maximization of an effectiveness ratio? trading off accuracy and computational cost. The unifying characteristic of the approaches is that all can be viewed as maximizing a penalized likelihood function. The second approach with suitable prior distributions has been shown to reduce to the first. This paper shows that the third approach reduces to the second for a particular form of the effectiveness ratio, and illustrates all three approaches with the problem of selecting the number of components in a mixture of Gaussian distributions. Unlike the first two approaches, the third can be used even when the candidate models are chosen for computational efficiency, without regard to physical interpretation, so that the likelihood and the prior distribution over models cannot be interpreted literally. As the most general and computationally oriented of the approaches, it is especially useful for artificial intelligence applications.