Two tales for a geometric Jensen--Shannon divergence

TL;DR

This paper introduces an extended geometric Jensen--Shannon divergence for positive measures, providing explicit formulas, theoretical properties, and applications to Gaussian distributions, enhancing divergence measures in information geometry.

Contribution

It proposes an alternative extended G-JSD for positive measures, analyzes its properties, and derives closed-form formulas for Gaussian cases, broadening divergence tools in information theory.

Findings

Extended G-JSD is an $f$-divergence with invariance properties.

Explicit formulas for Gaussian distributions are derived.

The divergence is not a metric, unlike the square root of JSD.

Abstract

The geometric Jensen--Shannon divergence (G-JSD) gained popularity in machine learning and information sciences thanks to its closed-form expression between Gaussian distributions. In this work, we introduce an alternative definition of the geometric Jensen--Shannon divergence tailored to positive densities which does not normalize geometric mixtures. This novel divergence is termed the extended G-JSD as it applies to the more general case of positive measures. We report explicitly the gap between the extended G-JSD and the G-JSD when considering probability densities, and show how to express the G-JSD and extended G-JSD using the Jeffreys divergence and the Bhattacharyya distance or Bhattacharyya coefficient. The extended G-JSD is proven to be a $f$-divergence which is a separable divergence satisfying information monotonicity and invariance in information geometry. We derive corresponding closed-form formula for the two types of G-JSDs when considering the case of multivariate Gaussian distributions often met in applications. We consider Monte Carlo stochastic estimations and approximations of the two types of G-JSD using the projective $\gamma$-divergences. Although the square root of the JSD yields a metric distance, we show that this is not anymore the case for the two types of G-JSD. Finally, we explain how these two types of geometric JSDs can be interpreted as regularizations of the ordinary JSD.