A Property of the Kullback--Leibler Divergence for Location-scale Models

TL;DR

This paper reveals a unique property of the Kullback--Leibler divergence between location-scale models, showing the divergence's minimum value is independent of the first model's parameters, with applications in Bayesian model selection.

Contribution

It demonstrates a novel property of KL divergence for location-scale models and extends it to models transformable into such distributions, with practical implications for Bayesian model selection.

Findings

Minimum KL divergence is independent of the first model's parameters.

Property holds for models transformable into location-scale distributions.

Application demonstrated in objective Bayesian model selection.

Abstract

In this paper, we discuss a property of the Kullback--Leibler divergence measured between two models of the family of the location-scale distributions. We show that, if model $M_1$ and model $M_2$ are represented by location-scale distributions, then the minimum Kullback--Leibler divergence from $M_1$ to $M_2$, with respect to the parameters of $M_2$, is independent from the value of the parameters of $M_1$. Furthermore, we show that the property holds for models that can be transformed into location-scale distributions. We illustrate a possible application of the property in objective Bayesian model selection.