On the Kullback-Leibler divergence between discrete normal distributions

TL;DR

This paper explores the properties of discrete normal distributions, focusing on formulas for statistical divergences like the Kullback-Leibler divergence, and introduces efficient approximation methods using advanced divergence measures.

Contribution

It provides new formulas for divergences between discrete normal distributions and proposes efficient approximation techniques leveraging Rényi and projective divergences.

Findings

Formulas for Kullback-Leibler divergence between discrete normal distributions

Efficient approximation methods using Rényi and projective divergences

Enhanced understanding of divergence computations on the integer lattice

Abstract

Discrete normal distributions are defined as the distributions with prescribed means and covariance matrices which maximize entropy on the integer lattice support. The set of discrete normal distributions form an exponential family with cumulant function related to the Riemann theta function. In this paper, we present several formula for common statistical divergences between discrete normal distributions including the Kullback-Leibler divergence. In particular, we describe an efficient approximation technique for calculating the Kullback-Leibler divergence between discrete normal distributions via the R\'enyi $\alpha$-divergences or the projective $\gamma$-divergences.