A Geometric View of Conjugate Priors

TL;DR

This paper provides a geometric interpretation of conjugate priors in Bayesian learning, framing them through Bregman divergence to enhance understanding and derive hyperparameters for hybrid models.

Contribution

It introduces a geometric perspective on conjugate priors using Bregman divergence, offering new insights and methods for hyperparameter derivation in semi-supervised learning.

Findings

Conjugate priors can be viewed as geometric entities defined by Bregman divergence.

Hyperparameters correspond to effective sample points in the geometric space.

The approach aids in deriving priors for hybrid generative-discriminative models.

Abstract

In Bayesian machine learning, conjugate priors are popular, mostly due to mathematical convenience. In this paper, we show that there are deeper reasons for choosing a conjugate prior. Specifically, we formulate the conjugate prior in the form of Bregman divergence and show that it is the inherent geometry of conjugate priors that makes them appropriate and intuitive. This geometric interpretation allows one to view the hyperparameters of conjugate priors as the {\it effective} sample points, thus providing additional intuition. We use this geometric understanding of conjugate priors to derive the hyperparameters and expression of the prior used to couple the generative and discriminative components of a hybrid model for semi-supervised learning.