The statistical Minkowski distances: Closed-form formula for Gaussian Mixture Models

TL;DR

This paper introduces new statistical Minkowski distances with closed-form formulas for Gaussian mixtures and other exponential family mixtures, enabling efficient computation of distances and diversity indices in probabilistic models.

Contribution

It extends Minkowski distances to probability densities, providing closed-form formulas for Gaussian mixtures and other exponential family mixtures, which was not previously available.

Findings

Closed-form formulas for Gaussian mixture distances.

Extension to mixtures of exponential families.

Introduction of Minkowski's diversity index.

Abstract

The traditional Minkowski distances are induced by the corresponding Minkowski norms in real-valued vector spaces. In this work, we propose novel statistical symmetric distances based on the Minkowski's inequality for probability densities belonging to Lebesgue spaces. These statistical Minkowski distances admit closed-form formula for Gaussian mixture models when parameterized by integer exponents. This result extends to arbitrary mixtures of exponential families with natural parameter spaces being cones: This includes the binomial, the multinomial, the zero-centered Laplacian, the Gaussian and the Wishart mixtures, among others. We also derive a Minkowski's diversity index of a normalized weighted set of probability distributions from Minkowski's inequality.

Equations

$$M_{\alpha }(x,y):=\Vert x-y{\Vert }_{\alpha }={\left(\sum _{i=1}^d|x_i-y_i{|}^{\alpha }\right)}^{\frac{1}{\alpha }},$$

$$L_{\alpha }(\mu ):=\left\{f\in \mathbb{F}:{\int }_{\mathcal{X}}|f(x){|}^{\alpha }\mathrm{d}\mu (x)<\infty \right\}.$$

$$M_{\alpha }(p,q):={\left({\int }_{\mathcal{X}}|p(x)-q(x){|}^{\alpha }\mathrm{d}\mu (x)\right)}^{\frac{1}{\alpha }}.$$

$$\mathrm{TV}(p,q):=\frac{1}{2}\int |p(x)-q(x)|\mathrm{d}\mu (x)=\frac{1}{2}\Vert p-q{\Vert }_{L_1(\mu )}=\frac{1}{2}{M}_1(p,q).$$

$${\left(\int |p(x)+q(x){|}^{\alpha }\mathrm{d}\mu (x)\right)}^{\frac{1}{\alpha }}\le {\left(\int |p(x){|}^{\alpha }\mathrm{d}\mu (x)\right)}^{\frac{1}{\alpha }}+{\left(\int |q(x){|}^{\alpha }\mathrm{d}\mu (x)\right)}^{\frac{1}{\alpha }},$$

$$\Vert p+q{\Vert }_{\alpha }\le \Vert p{\Vert }_{\alpha }+\Vert q{\Vert }_{\alpha }.$$

$$D_{\alpha }(p,q):=\Vert p{\Vert }_{\alpha }+\Vert q{\Vert }_{\alpha }-\Vert p+q{\Vert }_{\alpha }\ge 0.$$

$$L_{\alpha }(p,q):=-\log \frac{\Vert p+q{\Vert }_{\alpha }}{\Vert p{\Vert }_{\alpha }+\Vert q{\Vert }_{\alpha }}=\log \frac{\Vert p{\Vert }_{\alpha }+\Vert q{\Vert }_{\alpha }}{\Vert p+q{\Vert }_{\alpha }}\ge 0.$$

$$I(p_1,\dots ,p_k;{\alpha }_1,\dots ,{\alpha }_k):={\int }_{\mathcal{X}}{p}_1(x{)}^{{\alpha }_1}\dots p_k(x{)}^{{\alpha }_k}\mathrm{d}\mu (x),\alpha \in {\mathbb{R}}_{+}^k.$$

$$p_{\theta }(x)\propto \exp (t(x{)}^{\top }\theta ),$$

$$p_{\theta }(x)=\exp \left(t(x{)}^{\top }\theta -F(\theta )\right),$$

$$F(\theta ):=\log {\int }_{\mathcal{X}}\exp (t(x{)}^{\top }\theta )\mathrm{d}\mu (x),$$

$$\Theta :=\left\{\theta \in {\mathbb{R}}^D:{\int }_{\mathcal{X}}\exp (t(x{)}^{\top }\theta )\mathrm{d}\mu (x)<\infty \right\}.$$

$$I(p_{{\theta }_1},\dots ,p_{{\theta }_k};{\alpha }_1,\dots ,{\alpha }_k)=\exp \left(F\left(\sum _{i=1}^k{\alpha }_i{\theta }_i\right)-\sum _{i=1}^k{\alpha }_{i}F({\theta }_i)\right)<\infty ,$$

$$I(p_{{\theta }_1},\dots ,p_{{\theta }_k};{\alpha }_1,\dots ,{\alpha }_k)=\int \prod _{i=1}^k{\left(\exp \left(\left(t(x{)}^{\top }{\theta }_i-F({\theta }_i)\right)\right)\right)}^{{\alpha }_i}\mathrm{d}\mu (x),$$

$$J_F({\theta }_1,\dots ,{\theta }_k;{\alpha }_1,\dots ,{\alpha }_k):=\sum _i{\alpha }_{i}F({\theta }_i)-F\left(\sum _i{\alpha }_i{\theta }_i\right).$$

$$I(p_{{\theta }_1},\dots ,p_{{\theta }_k};{\alpha }_1,\dots ,{\alpha }_k)=\exp \left(-J_F({\theta }_1,\dots ,{\theta }_k;{\alpha }_1,\dots ,{\alpha }_k)\right)$$

$$\tilde{m}(x{)}^{\alpha }=\sum _{\begin{array}{c}{\sum }_{i=1}^k{\alpha }_i=\alpha \\ {\alpha }_i\in \mathbb{N}\end{array}}(\genfrac{}{}{0ex}{}{\alpha }{{\alpha }_1,\dots ,{\alpha }_k})\prod _{j=1}^k(w_j{p}_j(x){)}^{{\alpha }_j},$$

$$\left(\genfrac{}{}{0ex}{}{\alpha }{{\alpha }_1,\dots ,{\alpha }_k}\right):=\frac{\alpha !}{{\alpha }_1!\times \dots \times {\alpha }_k!},$$

$$\int \tilde{m}(x{)}^{\alpha }\mathrm{d}\mu (x)=\sum _{\begin{array}{c}{\sum }_i{\alpha }_i=\alpha \\ {\alpha }_i\in \mathbb{N}\end{array}}(\genfrac{}{}{0ex}{}{\alpha }{{\alpha }_1,\dots ,{\alpha }_k})\left(\prod _{j=1}^k{w}_j^{{\alpha }_j}\right)I(p_1,\dots ,p_k;{\alpha }_1,\dots ,{\alpha }_k).$$

$$\Vert \tilde{m}{\Vert }_{L_{\alpha }(\mu )}={\left(\sum _{\begin{array}{c}{\sum }_i{\alpha }_i=\alpha \\ {\alpha }_i\in \mathbb{N}\end{array}}\left(\genfrac{}{}{0ex}{}{\alpha }{{\alpha }_1,\dots ,{\alpha }_k}\right)\left(\prod _{j=1}^k{w}_j^{{\alpha }_j}\right)\exp \left(-J_F({\theta }_1,\dots ,{\theta }_k;{\alpha }_1,\dots ,{\alpha }_k)\right)\right)}^{\frac{1}{\alpha }},$$

$$\left(\genfrac{}{}{0ex}{}{\alpha }{{\alpha }_1,\dots ,{\alpha }_k}\right)=\sum _{i=1}^k\left(\genfrac{}{}{0ex}{}{\alpha -1}{{\alpha }_1,\dots ,{\alpha }_i-1,\dots ,{\alpha }_k}\right),$$

$${\left(\sum _{i=1}^k{x}_i\right)}^{\alpha }=\sum _{{\alpha }_1=0}^{\alpha }\sum _{{\alpha }_2=0}^{{\alpha }_1}\dots \sum _{{\alpha }_{k-1}=0}^{{\alpha }_{k-2}}\left(\genfrac{}{}{0ex}{}{\alpha }{{\alpha }_1}\right)\left(\genfrac{}{}{0ex}{}{{\alpha }_1}{{\alpha }_2}\right)\dots \left(\genfrac{}{}{0ex}{}{{\alpha }_{k-1}}{{\alpha }_{k-2}}\right)x_1^{\alpha -{\alpha }_1}{x}_2^{{\alpha }_1-{\alpha }_2}\dots x_{k-1}^{{\alpha }_{k-2}-{\alpha }_{k-1}}{x}_k^{{\alpha }_{k-1}}.$$

$$M_{\alpha }(m,m^{\prime })$$

$$\tilde{m}(x)$$

$$=$$

$$(x_1+x_2{)}^{\alpha }=\sum _{i=0}^{\infty }(\genfrac{}{}{0ex}{}{\alpha }{i})x_1^{\alpha -i}{x}_2^i,$$

$$\left(\genfrac{}{}{0ex}{}{\alpha }{i}\right):=\frac{\alpha (\alpha -1)\dots (\alpha -i+1)}{i!}=\frac{\Gamma (\alpha +1)}{\Gamma (\alpha -i+1)\Gamma (i+1)},$$

$$p(x;{\lambda }_1,\dots ,{\lambda }_d)=\prod _{i=1}^d{\lambda }_i^{x_i},$$

$$p(x;\mu ,\Sigma )=\frac{1}{(2\pi {)}^{d/2}|\Sigma {|}^{1/2}}\exp \left(-\frac{(x-\mu {)}^T{\Sigma }^{-1}(x-\mu )}{2}\right),x\in {\mathbb{R}}^d$$

Full text

Keywords: Minkowski $\ell_{p}$ metrics, $L_{p}$ spaces, Minkowski’s inequality, statistical mixtures, exponential families, multinomial theorem, statistical divergence, information radius, projective distance, scale-invariant distance, homogeneous distance.

1 Introduction and motivation

1.1 Statistical distances between mixtures

Gaussian Mixture Models (GMMs) are flexible statistical models often used in machine learning, signal processing and computer vision [41, 19] since they can arbitrarily closely approximate any smooth density. To measure the dissimilarity between probability distributions, one often relies on the principled information-theoretic Kullback-Leibler (KL) divergence [8], commonly called the relative entropy. However the lack of closed-form formula for the KL divergence between GMMs111When the GMMs share the same components, it is known that the KL divergence between them amount to an equivalent Bregman divergence [35] that is however computationally intractable because its corresponding Bregman generator is the differential negentropy that does not admit a closed-form expression in that case. has motivated various KL lower and upper bounds [16, 15, 37, 38] for GMMs or approximation techniques [10], and further spurred the design of novel distances that admit closed-form formula between GMMs [28]. To give a few examples, let us cite the statistical squared Euclidean distance [19, 21], the Jensen-Rényi divergence [41] (for the quadratic Rényi entropy), the Cauchy-Schwarz (CS) divergence [18, 20], and a statistical distance based on discrete optimal transport [22, 38].

A distance $D:\mathcal{X}\times\mathcal{X}\rightarrow\mathbb{R}$ is a non-negative real-valued function $D$ on the product space $\mathcal{X}\times\mathcal{X}$ such that $D(p,q)=D((p,q))=0$ iff. $p=q$. A distance $D(p:q)$ between $p$ and $q$ may not be symmetric: This fact is emphasized by the ’:’ delimiter notation: $D(p:q)\not=D(q:p)$. For example, the KL divergence is an oriented distance: $\mathrm{KL}(p:q)\not=\mathrm{KL}(q:p)$. Two usual symmetrizations of the KL divergence are the Jeffreys’ divergence and the Jensen-Shannon divergence [27]. Informally speaking, a divergence222Also called a contrast function in [11]. is a smooth distance333A Riemannian distance is not smooth but a squared Riemannian distance is smooth. that allows one to define an information-geometric structure [2]. In other words, a divergence is a smooth premetric distance [9].

Recently, the Cauchy-Schwarz divergence [18] has been generalized to Hölder divergences [39]. These Cauchy and Hölder distances $D(p:q)$ are said to be projective because $D(\lambda p:\lambda^{\prime}q)=D(p:q)$ for any $\lambda,\lambda^{\prime}>0$. An important family of projective divergences for robust statistical inference are the $\gamma$-divergences [13, 33]. Interestingly, those projective distances do not require to handle normalized probability densities but only need to consider positive densities instead (handy in applications). The Hölder projective divergences do not admit closed-form formula for GMMs, except for the very special case of the CS divergence. The underlying reason is that the conjugate exponents $\frac{1}{\alpha}+\frac{1}{\beta}=1$ of Hölder divergences would need to be both integers. This constraint yields $\alpha=\beta=1$, giving the special case of the CS divergence (the other integer exponent case is in the limit when $\alpha=0$ and $\beta=\infty$).

1.2 Minkowski distances and Lebesgue spaces

The renown Minkowski distances are norm-induced metrics [9] measuring distances between $d$-dimensional vectors $x,y\in\mathbb{R}^{d}$ defined for $\alpha\geq 1$ by:

[TABLE]

where the Minkowski norms are given by $\|x\|_{\alpha}=\left(\sum_{i=1}^{d}|x_{i}|^{\alpha}\right)^{\frac{1}{\alpha}}$. The Minkowski norms can be extended to countably infinite-dimensional $\ell_{\alpha}$ spaces of sequences (see [1], p. 68).

Let $(\mathcal{X},\mathcal{F})$ be a measurable space where $\mathcal{F}$ denotes the $\sigma$-algebra of $\mathcal{X}$, and let $\mu$ be a probability measure (with $\mu(\mathcal{X})=1$) with full support $\mathrm{supp}(\mu)=\mathcal{X}$ (where $\mathrm{supp}(\mu){:=}\mathrm{cl}(\{F\in\mathcal{F}{\ :\ }\mu(F)>0\})$ and $\mathrm{cl}$ denotes the set closure). Let $\mathbb{F}$ be the set of all real-valued measurable functions defined on $\mathcal{X}$. We define the Lebesgue space [1] $L_{\alpha}(\mu)$ for $\alpha\geq 1$ as follows:

[TABLE]

The Minkowski distance [25] of Eq. 1 can be generalized to probability densities belonging to Lebesgue $L_{\alpha}(\mu)$ spaces, to get the statistical Minkowski distance for $\alpha\geq 1$:

[TABLE]

When $\alpha=1$, we recover twice the Total Variation (TV) metric:

[TABLE]

Notice that the statistical Minkowski distance does not admit closed-form formula in general because of the absolute value. The total variation is related to the probability of error in Bayesian statistical hypothesis testing [29].

In this work, we design novel distances based on the Minkowski’s inequality (triangle inequality for $L_{\alpha}(\mu)$, which proves that $\|p\|_{L_{\alpha}(\mu)}$ is a norm (i.e., the $L_{\alpha}$-norm), so that the statistical Minkowski’s distance between functions of a Lebesgue space can be written as $M_{\alpha}(p,q)=\|p-q\|_{L_{\alpha}(\mu)}$). The space $L_{\alpha}(\mu)$ is a Banach space (ie., complete normed linear space).

1.3 Paper outline

The paper is organized as follows: Section 2 defines the new Minkowski distances by measuring in various ways the tightness of the Minkowski’s inequality applied to probability densities. Section 3 proves that all these statistical Minkowski distances admit closed-form formula for mixture of exponential families with conic natural parameter spaces for integer exponents. In particular, this includes the case of Gaussian mixture models. Section 4 lists a few examples of common exponential families with conic natural parameter spaces. In Section 5, we define Minkowski’s diversity indices for a normalized weighted set of probability distributions. Finally, section 6 concludes this work and hints at perspectives.

2 Distances from the Minkowski’s inequality

Let us state Minkowski’s inequality:

Theorem 1** (Minkowski’s inequality).**

For $p(x),q(x)\in L_{\alpha}(\mu)$ with $\alpha\in[1,\infty)$, we have the following Minkowski’s inequality:

[TABLE]

with equality holding only when $q(x)=0$ (almost everywhere, a.e.), or when $p(x)=\lambda q(x)$ a.e. for $\lambda>0$ for $\alpha>1$.

The usual proof of Minkowski’s inequality relies on Hölder’s inequality [40, 39]. Following [39], we define distances by measuring in several ways the tightness of the Minkowski’s inequality. When clear from context, we shall write $\|\cdot\|_{\alpha}$ for short instead of $\|\cdot\|_{L_{\alpha}(\mu)}$. Thus Minkowski’s inequality writes as:

[TABLE]

Minkowski’s inequality proves that the $L_{\alpha}$-spaces are normed vector spaces.

Notice that when $p(x)$ and $q(x)$ are probability densities (i.e., $\int p(x)\mathrm{d}\mu(x)=\int q(x)\mathrm{d}\mu(x)=1$), Minkowski’s inequality becomes an equality iff. $p(x)=q(x)$ almost everywhere, for $\alpha>1$. Thus we can define the following novel Minkowski’s distances between probability densities satisfying the identity of indiscernibles:

Definition 2** (Minkowski difference distance).**

For probability densities $p,q\in L_{\alpha}(\mu)$, we define the Minkowski difference $D_{\alpha}(\cdot,\cdot)$ distance for $\alpha\in(1,\infty)$ as:

[TABLE]

Definition 3** (Minkowski log-ratio distance).**

For probability densities $p,q\in L_{\alpha}(\mu)$, we define the Minkowski log-ratio distance $L_{\alpha}(\cdot,\cdot)$ for $\alpha\in(1,\infty)$ as:

[TABLE]

By construction, all these Minkowski distances are symmetric distances: Namely, $M_{\alpha}(p,q)=M_{\alpha}(q,p)$, $D_{\alpha}(p,q)=D_{\alpha}(q,p)$ and $L_{\alpha}(p,q)=L_{\alpha}(q,p)$.

Notice that $L_{\alpha}(p,q)$ is scale-invariant444Like any distance based on the log ratio of triangle inequality gap induced by a homogeneous norm.: $L_{\alpha}(\lambda p,\lambda q)=L_{\alpha}(p,q)$ for any $\lambda>0$. Scale-invariance is a useful property in many signal processing applications. For example, the scale-invariant Itakura-Saito divergence (a Bregman divergence) has been successfully used in Nonnegative Matrix Factorization [12] (NMF). Distance $D_{\alpha}(p,q)$ is homogeneous since $D_{\alpha}(\lambda p,\lambda q)=|\lambda|D_{\alpha}(p,q)$ for any $\lambda\in\mathbb{R}$ (and so is distance $M_{\alpha}(p,q)$).

3 Closed-form formula for statistical mixtures of exponential families

In this section, we shall prove that $D_{\alpha}$ and $L_{\alpha}$ between statistical mixtures are in closed-form for all integer exponents (and $M_{\alpha}$ for all even exponents) for mixtures of exponential families with conic natural parameter spaces.

Let us first define the positively weighed geometric integral $I$ of a set $\{p_{1},\ldots,p_{k}\}$ of $k$ probability densities of $L_{\alpha}(\mu)$ as:

[TABLE]

An exponential family [7, 31] $\mathcal{E}_{t,\mu}$ is a set $\{p_{\theta}(x)\}_{\theta}$ of probability densities wrt. $\mu$ which densities can be expressed proportionally canonically as:

[TABLE]

where $t(x)$ is a $D$-dimensional vector of sufficient statistics [7]. The term $t(x)^{\top}\theta$ can be written equivalently as ${\langle t(x),\theta\rangle}$, where ${\langle\cdot,\cdot\rangle}$ denotes the scalar product on $\mathbb{R}^{D}$. Thus the normalized probability densities of $\mathcal{E}_{t,\mu}$ can be written as:

[TABLE]

where

[TABLE]

is called the log-partition function (also called cumulant function [7] or log-normalizer [31]). The natural parameter space is:

[TABLE]

Many common distributions (Gaussians, Poisson, Beta, etc.) belong to exponential families in disguise [7, 31].

Lemma 4**.**

For probability densities $p_{\theta_{1}},\ldots,p_{\theta_{k}}$ belonging to the same exponential family $\mathcal{E}_{t,\mu}$, we have:

[TABLE]

provided that $\sum_{i=1}^{k}\alpha_{i}\theta_{i}\in\Theta$.

Proof.

[TABLE]

since $\int_{\mathcal{X}}\exp\left(t(x)^{\top}(\sum_{i}\alpha_{i}\theta_{i})-F(\sum_{i}\alpha_{i}\theta_{i})\right)\mathrm{d}\mu(x)=\int_{\mathcal{X}}p_{\sum_{i}\alpha_{i}\theta_{i}}(x)\mathrm{d}\mu(x)=1$, provided that $\bar{\theta}{:=}\sum_{i}\alpha_{i}\theta_{i}\in\Theta$ (and $p_{\bar{\theta}}\in\mathcal{E}_{t,\mu}$). ∎

In particular, the condition $\sum_{i}\alpha_{i}\theta_{i}\in\Theta$ always holds when the natural parameter space $\Theta$ is a cone. In the remainder, we shall call those exponential families with natural parameter space being a cone, Conic Exponential Families (CEFs) for short. Note that when $\sum_{i}\alpha_{i}\theta_{i}\not\in\Theta$, the integral $I(p_{\theta_{1}},\ldots,p_{\theta_{k}};\alpha_{1},\ldots,\alpha_{k})$ diverges (that is, $I(p_{\theta_{1}},\ldots,p_{\theta_{k}};\alpha_{1},\ldots,\alpha_{k})=\infty$).

Observe that for a CEF density $p_{\theta}(x)$, we have $p_{\theta}(x)^{\alpha}$ in $L_{\alpha}(\mu)$ for any $\alpha\in[1,\infty)$.

Corollary 5**.**

We have $I(p_{\theta_{1}},\ldots,p_{\theta_{k}};\alpha_{1},\ldots,\alpha_{k})=\exp\left(F\left(\sum_{i}\alpha_{i}\theta_{i}\right)-\sum_{i}\alpha_{i}F(\theta_{i})\right)<\infty$ for probability densities belonging to the same exponential family with natural parameter space $\Theta$ being a cone.

We also note in passing that $I(p_{1},\ldots,p_{k};\alpha_{1},\ldots,\alpha_{k})<\infty$ for $\alpha\in\mathbb{R}^{k}$ for probability densities belonging to the same exponential family with natural parameter space being an affine space (e.g., Poisson or isotropic Gaussian families [32]).

Let us define:

[TABLE]

This quantity is called the Jensen diversity [30] when $\alpha\in\Delta_{k}$ (the $(k-1)$-dimensional standard simplex), or Bregman information555Because $\sum_{i}\alpha_{i}B_{F}(\theta_{i}:\bar{\theta})=J_{F}({\theta_{1}},\ldots,{\theta_{k}};\alpha_{1},\ldots,\alpha_{k})$ for the barycenter $\bar{\theta}=\sum_{i}\alpha_{i}\theta_{i}$, where $B_{F}(\theta:\theta^{\prime})=F(\theta)-F(\theta^{\prime})-(\theta-\theta^{\prime})^{\top}\nabla F(\theta^{\prime})$ is a Bregman divergence. in [5]. Although the Jensen diversity is non-negative when $\alpha\in\Delta_{k}$, this Jensen diversity of Eq. 15 maybe negative when $\alpha\in\mathbb{R}_{+}^{k}$. When $\alpha\in\mathbb{R}_{+}^{k}$, we thus call the Jensen diversity the generalized Jensen diversity. It follows that we have:

[TABLE]

The CEFs include the Gaussian family, the Wishart family, the Binomial/multinomial family, etc. [7, 31, 28].

Let us consider a finite positive mixture $\tilde{m}(x)=\sum_{i=1}^{k}w_{i}p_{i}(x)$ of $k$ probability densities, where the weight vector $w\in\mathbb{R}_{+}^{k}$ are not necessarily normalized to one.

Lemma 6**.**

For a finite positive mixture $\tilde{m}(x)$ with components belonging to the same CEF, $\|\tilde{m}\|_{L_{\alpha}(\mu)}$ is finite and in closed-form, for any integer $\alpha\geq 2$.

Proof.

Consider the multinomial expansion $\tilde{m}(x)^{\alpha}$ obtained by applying the multinomial theorem [6]:

[TABLE]

where

[TABLE]

is the multinomial coefficient [4]. It follows that:

[TABLE]

Thus the term $\int\tilde{m}(x)^{\alpha}\mathrm{d}\mu(x)$ amounts to a positively weighted sum of integrals of monomials that are positively weighted geometric means of mixture components. When $p_{i}=p_{\theta_{i}}$, since $I(p_{\theta_{1}},\ldots,p_{\theta_{k}};\alpha_{1},\ldots,\alpha_{k})<\infty$ using Eq. 5, we conclude that $\tilde{m}\in{L_{\alpha}(\mu)}$ for $\alpha\in\mathbb{N}$, and we get the formula:

[TABLE]

for $\alpha\in\mathbb{N}$. ∎

A naive multinomial expansion of $\tilde{m}(x)^{\alpha}$ yields $k^{\alpha}$ terms that can then be simplified. Using the multinomial theorem, there are $\binom{k+\alpha-1}{\alpha}$ integral terms in the formula of $\int(\sum_{i=1}^{k}w_{i}p_{i}(x))^{\alpha}\mathrm{d}\mu(x)$. This number corresponds to the number of sequences of $k$ disjoint subsets whose union is $\{1,\ldots,\alpha\}$ (also called the number of ordered partitions but beware that some sets may be empty).

The multinomial expansion can be calculated efficiently using a generalization of Pascal’s triangle, called Pascal’s simplex [26], thus avoiding to compute from scratch all the multinomial coefficients.

We have the following generalized Pascal’s recurrence formula for calculating the multinomial coefficients:

[TABLE]

with the terminal cases $\binom{\alpha}{\alpha_{1},\ldots,\alpha_{k}}=0$ if there exists an $\alpha_{i}<0$. Also by convention, we set conveniently $\binom{\alpha}{\alpha_{1},\ldots,\alpha_{k}}=0$ if there exists $\alpha_{i}>\alpha$.

An efficient way to implement the multinomial expansion using nested iterative loops follows from this identity:

[TABLE]

We are now ready to show when the statistical Minkowski’s distances $M_{\alpha},D_{\alpha}$ and $L_{\alpha}$ are in closed-form for mixtures of CEFs using Lemma 6.

Theorem 7** (Closed-form formula for Minkowski’s distances).**

For mixtures $m=\sum_{i=1}^{k}w_{i}p_{\theta_{i}}$ and $m^{\prime}=\sum_{j=1}^{k^{\prime}}w_{j}^{\prime}p_{\theta_{j}^{\prime}}$ of CEFs $\mathcal{E}_{\mu,t}$, $D_{\alpha}$ and $L_{\alpha}$ admits closed-form formula for integers $\alpha\geq 2$, and $M_{\alpha}$ is in closed-form when $\alpha\geq 2$ is an even positive integer.

Proof.

For $D_{\alpha}$ and $L_{\alpha}$, it is enough to show that $\|m\|_{L_{\alpha}(\mu)},\|m^{\prime}\|_{L_{\alpha}(\mu)}$ and $\|m+m^{\prime}\|_{L_{\alpha}(\mu)}$ are all in closed-form. This follows from Lemma 6 by setting $\tilde{m}$ to be $m$, $m^{\prime}$ and $m+m^{\prime}$, respectively. The overall number of generalized Jensen diversity terms in the formula of $D_{\alpha}$ or $L_{\alpha}$ is $O\left(\binom{k+k^{\prime}+\alpha-1}{\alpha}\right)$.

Now, consider distance $M_{\alpha}$. To get rid of the absolute value in $M_{\alpha}$ for even integers $\alpha$, we rewrite $M_{\alpha}$ as follows:

[TABLE]

Let $\tilde{m}(x)=(m(x)-m^{\prime}(x))^{2}$. We have:

[TABLE]

We have the density products $p_{\theta,\theta^{\prime}}{:=}p_{\theta}p_{\theta^{\prime}}=I(p_{\theta},p_{\theta^{\prime}};1,1)\in L_{\frac{\alpha}{2}}(\mu)$ (using Lemma 6) for any $\theta,\theta^{\prime}\in\Theta$ and $\alpha\geq 2$. When $\alpha=2$, $\frac{\alpha}{2}=1$, and we easily reach a closed-form formula for $M_{2}(m,m^{\prime})$. Otherwise, let us expand all the terms in Eq. 25, and rewrite $\tilde{m}(x)=\sum_{l=1}^{K}w_{l}^{\prime\prime}p_{\theta_{l},\theta^{\prime}_{l}}$. Now, a key difference is that $w_{l}^{\prime\prime}\in\mathbb{R}$, and not necessarily positive. Nevertheless, since $\frac{\alpha}{2}\in\mathbb{N}$, we can still use the multinomial theorem to expand $\tilde{m}(x)^{\frac{\alpha}{2}}$, distribute the integral over all terms, and compute elementary integrals $I(p_{\theta_{1},\theta^{\prime}_{1}},\ldots,p_{\theta_{K},\theta^{\prime}_{K}};\alpha_{1}^{\prime},\ldots,\alpha_{K}^{\prime})$ with $\sum_{l=1}^{K}\alpha_{i}^{\prime}=\frac{\alpha}{2}$ in closed-form. Thus $M_{\alpha}$ is available in closed-form for mixtures of CEFs for all even positive integers $\alpha\geq 2$. The number of terms in the $M_{\alpha}$ formula is $O\left(\binom{\max(k^{2},{k^{\prime}}^{2})+\alpha-1}{\alpha}\right)$. ∎

Note that there exists a generalization666There also exists a generalization of the multinomial theorem to real exponents, however, this is much less known in the literature (see http://fractional-calculus.com/multinomial_theorem.pdf). of the binomial theorem to real exponents $\alpha\in\mathbb{R}$ called Newton’s generalized binomial theorem using an infinite series of general binomial coefficients:

[TABLE]

with the generalized binomial coefficient defined by:

[TABLE]

where $\Gamma(x){:=}\int_{0}^{\infty}t^{x-1}e^{-t}\mathrm{d}t$ is the Gamma function extending the factorial: $\Gamma(n)=(n-1)!$. Equation 26 is only valid whenever the infinite series converge. That is, for $|x_{1}|\geq|x_{2}|$. When extending to mixture densities (i.e., $(w_{1}p_{1}(x)+w_{2}p_{2}(x))^{\alpha}$) and taking the integral, we therefore need to split the integral into two integrals depending on whether $w_{1}p_{1}(x)\geq w_{2}p_{2}(x)$, or not. Furthermore, we need to compute these integrals on truncated support domains: This becomes very tricky as the dimension of the support increase [14].

4 Some examples of conic exponential families

Let us report a few conic exponential families with their respective canonical decompositions. The measure $\mu$ is usually either the Lebesgue measure on the Euclidean space (i.e., $\mathrm{d}\mu(x)=\mathrm{d}x$), or the counting measure.

• •

Bernoulli/multinomial families. The Bernoulli density is $p(x;\lambda)=\lambda^{x}(1-\lambda)^{1-x}$ with $\lambda\in(0,1)=\Delta_{1}$, for $\mathcal{X}=\{0,1\}$. The natural parameter is $\theta=\log\frac{\lambda}{1-\lambda}$ and the conic natural parameter space is $\Theta=\mathbb{R}$. The log-partition function is $F(\theta)=\log(1+e^{\theta})$. The sufficient statistics is $t(x)=x$.

The multinomial density generalizes the Bernoulli and the binomial densities. Here, we consider the categorical distribution also called “multinoulli” distribution. The multinoulli density is given by:

[TABLE]

where $\lambda\in\Delta_{d}$, the $(d-1)$-dimensional standard simplex. We have $\mathcal{X}=\{0,1\}^{d}$. The sufficient statistic vector is $t(x)=(x_{1},\ldots,x_{d-1})$. The natural parameter is a $(d-1)$-dimensional vector with natural coordinates $\theta=\left(\log\frac{\lambda_{1}}{\lambda_{d}},\ldots,\log\frac{\lambda_{d-1}}{\lambda_{d}}\right)$. The conic natural parameter space is $\Theta=\mathbb{R}^{d-1}$ (ie., a non-pointed cone). The log-partition function is $F(\theta)=\log(1+\sum_{i=1}^{d-1}e^{\theta_{i}})$.

• •

Zero-centered Laplacian family. The density is $p(x;\sigma)=\frac{1}{2\sigma}e^{-\frac{|x|}{\sigma}}$ and the sufficient statistic is $t(x)=|x|$. The natural parameter is $\theta=-\frac{1}{\sigma}$ with the conic parameter space $\Theta=(-\infty,0)=\mathbb{R}_{--}$. The log-normalizer is $F(\theta)=\log(\frac{2}{-\theta})$. See [3] for an application of Laplacian mixtures.

• •

Multivariate Gaussian family. The probability density of a $d$-variante Gaussian distribution is:

[TABLE]

where $|\Sigma|$ denotes the determinant of the positive-definite matrix $\Sigma$. The natural parameter consists in a vector part $\theta_{v}$ and a matrix part $\theta_{M}$: $\theta=(\theta_{v},\theta_{M})=(\Sigma^{-1}\mu,\Sigma^{-1})$. The conic natural parameter space is $\Theta=\mathbb{R}^{d}\times S^{d}_{++}$, where $S^{d}_{++}$ denotes the cone of positive definite matrices of dimension $d\times d$. The sufficient statistics are $(x,xx^{\top})$. The log-partition function is:

[TABLE]

• •

Wishart family. The probability density is

[TABLE]

with $S\succ 0$ denoting the scale matrix and $n>d-1$ denoting the number of degrees of freedom, where $\Gamma_{d}$ is the multivariate Gamma function:

[TABLE]

${\mathrm{tr}}(X)$ denotes the trace of matrix $X$. The natural parameter is composed of a scalar $\theta_{s}$ and a matrix part $\theta_{M}$: $\theta=(\theta_{s},\theta_{M})=(\frac{n-d-1}{2},S^{-1})$. The conic natural parameter space is $\Theta=\mathbb{R}_{+}\times S^{d}_{++}$. The sufficient statistics are $(\log|X|,X)$. The log-partition function is:

[TABLE]

See [17] for an application of Wishart mixtures.

5 Minkowski’s diversity index

Informally speaking, a diversity index is a quantity that measures the variability of elements in a data set (i.e., the diversity of a population). For example, the (sample) variance of a (finite) point set is a diversity index. Point sets uniformly filling a large volume have large variance (and a large diversity index) while point sets with points concentrating to their centers of mass have low variance (and a small diversity index).

Recall that the Jensen diversity index [34] of a normalized weighted set $\{p_{1}=p_{\theta_{1}},\ldots,p_{n}=p_{\theta_{n}}\}$ of densities belonging to the same exponential family (also called information radius [23] or Bregman information [5, 36]) is defined for a strictly convex generator $F$ by:

[TABLE]

When $F(\theta)=\frac{1}{2}{\langle\theta,\theta\rangle}$, we recover from $J_{F}$ the variance.

We shall consider finite mixtures [24, 5] with linearly independent component densities. Using Minkowski’s inequality iteratively for $f_{1},\ldots,f_{n}\in L_{\alpha}(\mu)$, we get:

[TABLE]

When $\alpha>1$, equality holds when the $f_{i}$’s are proportional (a.e. $\mu$). By setting $f_{i}=w_{i}p_{i}$, we define the Minkowski’s diversity index:

Definition 8** (Minkowski’s diversity index).**

Define the Minkowski diversity index of $n$ weighted probability densities of $L_{\alpha}(\mu)$ for $\alpha>1$ by:

[TABLE]

It follows a closed-form formula for the Minkowski’s diversity index of a weighted set of distributions (ie., a mixture) belonging to the same CEF:

Corollary 9**.**

The Minkowski’s diversity index of $n$ weighted probability distributions belonging to the same conic exponential family is finite and admits a closed-form formula for any integer $\alpha\geq 2$.

6 Conclusion and perspectives

Designing novel statistical distances which admit closed-form formula for Gaussian mixture models is important for a wide range of applications in machine learning, computer vision and signal processing [18]. In this paper, we proposed to use the Minkowski’s inequality to design novel statistical symmetric Minkowski distances by measuring the tightness of the inequality either as an arithmetic difference or as a log-ratio of the left-hand-side and right-hand-side of the inequality. We showed that these novel statistical Minkowski distances yield closed-form formula for mixtures of exponential families with conic natural parameter spaces whenever the integer exponent $\alpha\geq 2$. In particular, this result holds for Gaussian mixtures, Bernoulli mixtures, Wishart mixtures, etc. We termed those families as Conic Exponential Families (CEFs). We also reported a closed-form formula for the ordinary statistical Minkowski distance for even positive integer exponents. Finally, we defined the Minkowski’s diversity index of a weighted population of probability distributions (a mixture), and proved that this diversity index admits a closed-form formula when the distributions belong to the same CEF.

Let us conclude by listing the formula of the statistical Minkowski distances for $\alpha=2$ for comparison with the Cauchy-Schwarz (CS) divergence:

[TABLE]

where ${\langle f,g\rangle}_{2}=\int f(x)g(x)\mathrm{d}\mu(x)$ for $f,g\in L_{2}(\mu)$. Note that for $\alpha=2$, $L_{2}(\mu)$ is a Hilbert space when equipped with this inner product. We get closed-form formula for these statistical Minkowski’s distances between mixtures $m_{1}$ and $m_{2}$ of CEFs, as well as for the Cauchy-Schwarz divergence. All those statistical distances can be computed in quadratic time in the number of mixture components.

Selecting a proper divergence from a priori first principles for a given application is a paramount but difficult task [9]. Often one is left by checking experimentally the performances of a few candidate divergences in order to select the a posteriori ‘best’ one. We hope that these newly proposed statistical Minkowski’s distances, $D_{\alpha}$ and scale-invariant $L_{\alpha}$, will prove experimentally useful in a number of applications ranging from computer vision to machine learning and signal processing.

Additional material is available from

https://franknielsen.github.io/MinkowskiStatDist/

Acknowledgments: The author would like to thank Gaëtan Hadjeres for his careful reading and feedback.