Kullback-Leibler Divergence for the Normal-Gamma Distribution

TL;DR

This paper derives the KL divergence for the normal-gamma distribution, revealing its equivalence to the Bayesian complexity penalty, and demonstrates its applications through simulated and real data.

Contribution

It provides the first derivation of the KL divergence for the normal-gamma distribution and links it to Bayesian model complexity.

Findings

KL divergence for normal-gamma is identical to Bayesian complexity penalty

Applications demonstrated on simulated data

Applications demonstrated on empirical data

Abstract

We derive the Kullback-Leibler divergence for the normal-gamma distribution and show that it is identical to the Bayesian complexity penalty for the univariate general linear model with conjugate priors. Based on this finding, we provide two applications of the KL divergence, one in simulated and one in empirical data.