On H\"older projective divergences

TL;DR

This paper introduces a new framework for constructing statistical divergences based on inequalities, specifically H"older inequalities, leading to novel, rescaling-invariant distances that outperform traditional divergences in clustering tasks.

Contribution

The paper develops two new classes of H"older divergences and pseudo-divergences, providing closed-form formulas for exponential family distributions and demonstrating their effectiveness in clustering.

Findings

New H"older divergences encapsulate Cauchy-Schwarz divergence

Closed-form formulas for exponential family distributions

H"older divergences outperform Cauchy-Schwarz in clustering

Abstract

We describe a framework to build distances by measuring the tightness of inequalities, and introduce the notion of proper statistical divergences and improper pseudo-divergences. We then consider the H\"older ordinary and reverse inequalities, and present two novel classes of H\"older divergences and pseudo-divergences that both encapsulate the special case of the Cauchy-Schwarz divergence. We report closed-form formulas for those statistical dissimilarities when considering distributions belonging to the same exponential family provided that the natural parameter space is a cone (e.g., multivariate Gaussians), or affine (e.g., categorical distributions). Those new classes of H\"older distances are invariant to rescaling, and thus do not require distributions to be normalized. Finally, we show how to compute statistical H\"older centroids with respect to those divergences, and carry out center-based clustering toy experiments on a set of Gaussian distributions that demonstrate empirically that symmetrized H\"older divergences outperform the symmetric Cauchy-Schwarz divergence.

Tables

Table 1. Table 1 : Clustering accuracy of the 2D Gaussian dataset (based on 500 independent runs).

Parameters	$\alpha =\gamma =1.1$	$\alpha =\gamma =1.5$	$\alpha =\gamma =2$ (CS)	$\alpha =\gamma =10$
$n=50$	$95.6\%\pm 7.6\%$	$92.4\pm 9.8\%$	$92.2\%\pm 9.9\%$	$92.2\%\pm 9.7\%$
$n=100$	$97.3\%\pm 5.3\%$	$94.9\pm 8.8\%$	$94.0\%\pm 10.1\%$	$94.2\%\pm 9.3\%$

Equations

$$\Delta =\mathrm{rhs}-\mathrm{lhs}\ge 0.$$

$$D=\log \frac{\mathrm{rhs}}{\mathrm{lhs}}=-\log \frac{\mathrm{lhs}}{\mathrm{rhs}}\ge 0.$$

$$\forall x\in \mathcal{X},$$

$$\displaystyle D(p,q)=\left\{\begin{array}[]{ll}\sum_{x\in\cal X}\big{[}\mathrm{rhs}(p(x),q(x))-\mathrm{lhs}(p(x),q(x))\big{]}&\mbox{discrete case},\\ \int_{\cal X}\big{[}\mathrm{rhs}(p(x),q(x))-\mathrm{lhs}(p(x),q(x))\big{]}\mathrm{d}x&\mbox{continuous case}.\end{array}\right.$$

$${\int }_{\mathcal{X}}p(x)q(x)\mathrm{d}x\le \sqrt{\left({\int }_{\mathcal{X}}p(x{)}^2\mathrm{d}x\right)\left({\int }_{\mathcal{X}}q(x{)}^2\mathrm{d}x\right)},$$

$$D(\lambda p(x):{\lambda }^{\prime }q(x))=D(p(x):q(x)),\forall \lambda ,{\lambda }^{\prime }>0.$$

$$D_{\gamma }(p(x),q(x))$$

$$S_{\gamma }(p(x),q(x))$$

$$\mathrm{KL}(p(x):q(x))={\int }_{\mathcal{X}}p(x)\log \frac{p(x)}{q(x)}\mathrm{d}x,$$

$${\int }_{\mathcal{X}}p(x)q(x)\mathrm{d}x\le {\left({\int }_{\mathcal{X}}p(x{)}^{\alpha }\mathrm{d}x\right)}^{\frac{1}{\alpha }}{\left({\int }_{\mathcal{X}}q(x{)}^{\beta }\mathrm{d}x\right)}^{\frac{1}{\beta }},$$

$${\int }_{\mathcal{X}}p(x)q(x)\mathrm{d}x\ge {\left({\int }_{\mathcal{X}}p(x{)}^{\alpha }\mathrm{d}x\right)}^{\frac{1}{\alpha }}{\left({\int }_{\mathcal{X}}q(x{)}^{\beta }\mathrm{d}x\right)}^{\frac{1}{\beta }}.$$

$$D_{\alpha }^{\mathtt{H}}(p(x):q(x))=-\log \left(\frac{{\int }_{\mathcal{X}}p(x)q(x)\mathrm{d}x}{{\left({\int }_{\mathcal{X}}p(x{)}^{\alpha }\mathrm{d}x\right)}^{\frac{1}{\alpha }}{\left({\int }_{\mathcal{X}}q(x{)}^{\beta }\mathrm{d}x\right)}^{\frac{1}{\beta }}}\right).$$

$$D_{\alpha }^{\mathtt{H}}(p(x):q(x))=\log \left(\frac{{\int }_{\mathcal{X}}p(x)q(x)\mathrm{d}x}{{\left({\int }_{\mathcal{X}}p(x{)}^{\alpha }\mathrm{d}x\right)}^{\frac{1}{\alpha }}{\left({\int }_{\mathcal{X}}q(x{)}^{\beta }\mathrm{d}x\right)}^{\frac{1}{\beta }}}\right).$$

$$D_2^{\mathtt{H}}(p(x):q(x))=\mathrm{CS}(p(x):q(x))=-\log \left(\frac{{\int }_{\mathcal{X}}p(x)q(x)\mathrm{d}x}{{\left({\int }_{\mathcal{X}}p(x{)}^2\mathrm{d}x\right)}^{\frac{1}{2}}{\left({\int }_{\mathcal{X}}q(x{)}^2\mathrm{d}x\right)}^{\frac{1}{2}}}\right),$$

$$D^{\mathtt{H}}_{\alpha}(p(x):p(x))=-\log\left(\frac{\int p(x)^{2}\mathrm{d}x}{\left(\int p(x)^{\alpha}\mathrm{d}x\right)^{\frac{1}{\alpha}}\left(\int p(x)^{\frac{\alpha}{\alpha-1}}\mathrm{d}x\right)^{\frac{\alpha-1}{\alpha}}}\right)\not=0\mbox{ when $\alpha\not=2$}.$$

$$-\log \left(\frac{\int p(x)\lambda p(x{)}^{\alpha -1}\mathrm{d}x}{{\left(\int p(x{)}^{\alpha }\mathrm{d}x\right)}^{\frac{1}{\alpha }}{\left(\int {\lambda }^{\beta }p(x{)}^{(\alpha -1)\beta }\mathrm{d}x\right)}^{\frac{1}{\beta }}}\right)=-\log \left(\frac{\int p(x{)}^{\alpha }\mathrm{d}x}{{\left(\int p(x{)}^{\alpha }\mathrm{d}x\right)}^{\frac{1}{\alpha }}{\left(\int p(x{)}^{\alpha }\mathrm{d}x\right)}^{\frac{1}{\beta }}}\right)=0,$$

$$D_{\alpha }^{\mathtt{H}}(p(x):q(x))=D_{\beta }^{\mathtt{H}}(q(x):p(x))=D_{\frac{\alpha }{\alpha -1}}^{\mathtt{H}}(q(x):p(x)).$$

$$S_{\alpha }^{\mathtt{H}}(p(x):q(x))$$

$$D_{\alpha }^{\mathtt{H}}(\tilde{p}(x):\tilde{q}(x))=D_{\alpha }^{\mathtt{H}}(p(x):q(x)),$$

$$D_{\alpha }^{\mathtt{H}}(\lambda p(x):{\lambda }^{\prime }q(x))=D_{\alpha }^{\mathtt{H}}(p(x):q(x))$$

$$p_{\alpha }^E(x)=\frac{p(x{)}^{\frac{1}{\alpha }}}{\int p(x{)}^{\frac{1}{\alpha }}\mathrm{d}x},$$

$$q_{\beta }^E(x)=\frac{q(x{)}^{\frac{1}{\beta }}}{\int q(x{)}^{\frac{1}{\beta }}\mathrm{d}x}.$$

$$D_{\alpha }^{\mathrm{HE}}(p(x):q(x))=D_{\alpha }^{\mathtt{H}}(p_{\alpha }^E(x):q_{\beta }^E(x))=-\log {\int }_{\mathcal{X}}p(x{)}^{1/\alpha }q(x{)}^{1-1/\alpha }\mathrm{d}x=B_{1/\alpha }(p(x):q(x)),$$

$${\mathrm{CS}}^{\mathrm{HE}}(p(x):q(x))=D_2^{\mathrm{HE}}(p(x):q(x))=D_2^H(p_2^E(x):q_2^E(x))=B_{1/2}(p(x):q(x))=B(p(x):q(x)).$$

$$D_{\alpha ,\gamma }^{\mathtt{H}}(p(x):q(x))=D_{\alpha }^{\mathtt{H}}(p(x{)}^{\gamma /\alpha }:q(x{)}^{\gamma /\beta })=-\log \left(\frac{{\int }_{\mathcal{X}}p(x{)}^{\gamma /\alpha }q(x{)}^{\gamma /\beta }\mathrm{d}x}{({\int }_{\mathcal{X}}p(x{)}^{\gamma }\mathrm{d}x{)}^{1/\alpha }({\int }_{\mathcal{X}}q(x{)}^{\gamma }\mathrm{d}x{)}^{1/\beta }}\right).$$

$$D_{\alpha ,\gamma }^{\mathtt{H}}(p(x):q(x))=D_{\bar{\alpha },\gamma }^{\mathtt{H}}(q(x),p(x)).$$

$$S_{\alpha ,\gamma }^{\mathtt{H}}(p:q)=\frac{D_{\alpha ,\gamma }^{\mathtt{H}}(p:q)+D_{\alpha ,\gamma }^{\mathtt{H}}(q:p)}{2}=-\log \sqrt{\frac{{\int }_{\mathcal{X}}p(x{)}^{\gamma /\alpha }q(x{)}^{\gamma /\beta }\mathrm{d}x{\int }_{\mathcal{X}}p(x{)}^{\gamma /\beta }q(x{)}^{\gamma /\alpha }\mathrm{d}x}{{\int }_{\mathcal{X}}p(x{)}^{\gamma }\mathrm{d}x{\int }_{\mathcal{X}}q(x{)}^{\gamma }\mathrm{d}x}}.$$

$$D_{\alpha ,\gamma }^{\mathtt{H}}(p(x):q(x))$$

$$=-\log {\left(p_{1/\gamma }^E(x)\right)}^{1/\alpha }{\left(q_{1/\gamma }^E(x)\right)}^{1/\beta }\mathrm{d}x=B_{\frac{1}{\alpha }}(p_{1/\gamma }^E(x):q_{1/\gamma }^E(x)).$$

$$D_{\alpha ,1}^{\mathtt{H}}(p(x):q(x))=-\log \left({\int }_{\mathcal{X}}p(x{)}^{1/\alpha }q(x{)}^{1/\beta }\mathrm{d}x\right)=B_{\frac{1}{\alpha }}(p(x):q(x)).$$

$$D_{2,2}^{\mathtt{H}}(p(x):q(x))=D_2^{\mathtt{H}}(p(x):q(x))=\mathrm{CS}(p(x):q(x)).$$

$$\int p(x)q(x)\mathrm{d}x\le \sqrt{\left(\int p(x{)}^2\mathrm{d}x\right)\left(\int p(x{)}^2\mathrm{d}x\right)},$$

Full text

Keywords: Hölder inequalities; Hölder divergences; projective divergences; Cauchy-Schwarz divergence; Hölder escort divergences; skew Bhattacharyya divergences; exponential families; conic exponential families; escort distribution; clustering.

1 Introduction: Inequality, proper divergence and improper pseudo-divergence

1.1 Statistical divergences from inequality gaps

An inequality is denoted mathematically by $\mathrm{lhs}\leq\mathrm{rhs}$, where $\mathrm{lhs}$ and $\mathrm{rhs}$ denote respectively the left-hand-side and right-hand-side of the inequality. One can build dissimilarity measures from inequalities $\mathrm{lhs}\leq\mathrm{rhs}$ by measuring the inequality tightness: For example, we may quantify the tightness of an inequality by its difference gap:

[TABLE]

When $\mathrm{lhs}>0$, the inequality tightness can also be gauged by the log-ratio gap:

[TABLE]

We may further compose this inequality tightness value measuring non-negative gaps with a strictly monotonically increasing function $f$ (with $f(0)=0$).

A bi-parametric inequality $\mathrm{lhs}(p,q)\leq\mathrm{rhs}(p,q)$ is called proper if it is strict for $p\neq{}q$ (i.e., $\mathrm{lhs}(p,q)<\mathrm{rhs}(p,q),\forall p\neq{}q$) and tight if and only if (iff) $p=q$ (i.e., $\mathrm{lhs}(p,q)=\mathrm{rhs}(p,q),\forall p=q$). Thus a proper bi-parametric inequality allows one to define dissimilarities such that $D(p,q)=0$ iff $p=q$. Such a dissimilarity is called proper. Otherwise, an inequality or dissimilarity is said improper. Note that there are many equivalent words used in the literature instead of (dis-)similarity: distance (although often assumed to have metric properties), pseudo-distance, discrimination, proximity, information deviation, etc.

A statistical dissimilarity between two discrete or continuous distributions $p(x)$ and $q(x)$ on a support $\mathcal{X}$ can thus be defined from inequalities by summing up or taking the integral for the inequalities instantiated on the observation space $\mathcal{X}$:

[TABLE]

In such a case, we get a separable divergence. Some non-separable inequalities induce a non-separable divergence. For example, the renown Cauchy-Schwarz divergence [8] is not separable because in the inequality:

[TABLE]

the rhs is not separable.

Furthermore, a proper dissimilarity is called a divergence in information geometry [2] when it is $C^{3}$ (i.e., three times differentiable thus allowing to define a metric tensor [34] and a cubic tensor [2]).

Many familiar distances can be reinterpreted as inequality gaps in disguise. For example, Bregman divergences [4] and Jensen divergences [28] (also called Burbea-Rao divergences [9, 22]) can be reinterpreted as inequality difference gaps, and the Cauchy-Schwarz distance [8] as an inequality log-ratio gap:

Example 1** (Bregman divergence as a Bregman score-induced gap divergence).**

A proper score function [14] $S(p:q)$ induces a gap divergence $D(p:q)=S(p:q)-S(p:p)\geq 0$. A Bregman divergence [4] $B_{F}(p:q)$ for a strictly convex and differentiable real-valued generator $F(x)$ is induced by the Bregman score $S_{F}(p:q)$. Let $S_{F}(p:q)=-F(q)-{\langle p-q,\nabla F(q)\rangle}$ denote the Bregman proper score minimized for $p=q$. Then the Bregman divergence is a gap divergence: $B_{F}(p:q)=S_{F}(p:q)-S_{F}(p:p)\geq 0$. When $F$ is strictly convex, the Bregman score is proper, and the Bregman divergence is proper.

Example 2** (Cauchy-Schwarz distance as a log-ratio gap divergence).**

Consider the Cauchy-Schwarz inequality $\int_{\mathcal{X}}p(x)q(x)\mathrm{d}x\leq\sqrt{\left(\int_{\mathcal{X}}p(x)^{2}\mathrm{d}x\right)\left(\int_{\mathcal{X}}q(x)^{2}\mathrm{d}x\right)}$. Then the Cauchy-Schwarz distance [8] between two continuous distributions is defined by ${\mathrm{CS}}(p(x):q(x))=-\log\frac{\int_{\mathcal{X}}p(x)q(x)\mathrm{d}x}{\sqrt{(\int_{\mathcal{X}}p(x)^{2}\mathrm{d}x)(\int_{\mathcal{X}}q(x)^{2}\mathrm{d}x)}}\geq 0$.

Note that we use the modern notation $D(p(x):q(x))$ to emphasize that the divergence is potentially asymmetric: $D(p(x):q(x))\not=D(q(x):p(x))$, see [2]. In information theory [11], the older notation “$||$” is often used instead of “$:$” that is used in information geometry [2].

To conclude this introduction, let us finally introduce the notion of projective statistical distances. A statistical distance $D(p(x):q(x))$ is said projective when:

[TABLE]

The Cauchy-Schwarz distance is a projective divergence. Another example of such a projective divergence is the parametric $\gamma$-divergence [13].

Example 3** ($\gamma$-divergence as a projective score-induced gap divergence).**

The $\gamma$-divergence [13, 29] $D_{\gamma}(p(x):q(x))$ for $\gamma>0$ is projective:

[TABLE]

The $\gamma$-divergence is related to the proper pseudo-spherical score [13].

The $\gamma$-divergences have been proven useful for robust statistical inference [13] in the presence of heavy outlier contamination.

1.2 Pseudo-divergences and the axiom of indiscernability

Consider a broader class of statistical pseudo-divergences based on improper inequalities, where the tightness of $\mathrm{lhs}(p,q)\leq\mathrm{rhs}(p,q)$ does not imply that $p=q$. This family of dissimilarity measures have interesting properties which have not been studied before.

Formally, statistical pseudo-divergences are defined with respect to density measures $p(x)$ and $q(x)$ with $x\in\mathcal{X}$, where $\mathcal{X}$ denotes the support. By definition, pseudo-divergences satisfy the following three fundamental properties:

Non-negativeness: $D(p(x):q(x))\geq 0$ for any $p(x),q(x)$; 2. 2.

Reachable indiscernability:

• •

$\forall{p(x)}$, there exists $q(x)$ such that $D(p(x):q(x))=0$,

• •

$\forall{q(x)}$, there exists $p(x)$ such that $D(p(x):q(x))=0$. 3. 3.

Positive correlation: If $D(p:q)=0$, then $\left(p(x_{1})-p(x_{2})\right)\left(q(x_{1})-q(x_{2})\right)\geq 0$ for any $x_{1}$, $x_{2}\in\mathcal{X}$.

As compared to statistical divergence measures such as the Kullback-Leibler (KL) divergence:

[TABLE]

pseudo-divergences do not require $D(p(x):p(x))=0$. Instead, any pair of distributions $p(x)$ and $q(x)$ with $D(p(x):q(x))=0$ only has to be “positively correlated” such that $p(x_{1})\leq{}p(x_{2})$ implies $q(x_{1})\leq{}q(x_{2})$, and vice versa. Any divergence with $D(p(x):q(x))=0\Rightarrow{}p(x)=q(x)$ (law of indiscernibles) automatically satisfies this weaker condition, and therefore any divergence belongs to the broader class of pseudo-divergences. Indeed, if $p(x)=q(x)$ then $(p(x_{1})-p(x_{2}))(q(x_{1})-q(x_{2}))=(p(x_{1})-p(x_{2}))^{2}\geq 0$. However the converse is not true. As we shall describe in the remainder, the family of pseudo-divergences is not limited to proper divergence measures. In the remainder, the term “pseudo-divergence” refers to such divergences that are not proper divergence measures.

We study two novel statistical dissimilarity families: One family of statistical improper pseudo-divergences and one family of proper statistical divergences. Within the class of pseudo-divergences, this work concentrates on defining a one-parameter family of dissimilarities called Hölder log-ratio gap divergence that we concisely abbreviate as HPD for “Hölder pseudo divergence” in the remainder. We also study its proper divergence counterpart termed HD for “Hölder divergence.”

1.3 Prior work and contributions

The term “Hölder divergence” has first been coined in 2014 based on the definition of the Hölder score [20, 19]: The score-induced Hölder divergence $D(p(x):q(x))$ is a proper gap divergence that yields a scale-invariant divergence [20, 19]. Let $p_{a,\sigma}(x)=a\sigma p(\sigma x)$ for $a,\sigma>0$ be a transformation. Then a scale-invariant divergence [20, 19] satisfies $D(p_{a,\sigma}(x):q_{a,\sigma}(x))=\kappa(a,\sigma)D(p(x):q(x))$ for a function $\kappa(a,\sigma)>0$. This gap divergence is proper since it is based on the so-called Hölder score [20, 19] but is not projective and does not include the Cauchy-Schwarz divergence. Due to these differences the Hölder log-ratio gap divergence introduced here shall not be confused with the Hölder gap divergence induced by the Hölder score [19, 20] that relies both on a scalar $\gamma$ and a function $\phi(\cdot)$.

We shall introduce two novel families of log-ratio projective gap divergences based on Hölder ordinary (or forward) and reverse inequalities that extend the Cauchy-Schwarz divergence, study their properties, and consider as an application clustering Gaussian distributions: We experimentally show better clustering results when using symmetrized Hölder divergences than using the Cauchy-Schwarz divergence. To contrast with the “Hölder composite score-induced divergences” of [19], our Hölder divergences admit closed-form expressions between distributions belonging to the same exponential families [23] provided that the natural parameter space is a cone or affine.

Our main contributions are summarized as follows:

• •

Define the uni-parametric family of Hölder improper pseudo-divergences (HPDs) in §2 and the bi-parametric family of Hölder proper divergences in §3 (HDs) for positive and probability measures, and study their properties (including their relationships with skewed Bhattacharrya distances [22] via escort distributions);

• •

Report closed-form expressions of those divergences for exponential families when the natural parameter space is a cone or affine (include but not limited to the cases of categorical distributions and multivariate Gaussian distributions) in §4;

• •

Provide approximation techniques to compute those divergences between mixtures based on log-sum-exp inequalities in §4.6;

• •

Describe a variational center-based clustering technique based on the convex-concave procedure for computing Hölder centroids, and report our experimental results in §5.

1.4 Organization

This paper is organized as follows: §2 introduces the definition and properties of Hölder pseudo-divergences (HPDs). It is followed by §3 that describes Hölder proper divergences (HDs). In §4, closed-form expressions for those novel families of divergences are reported for the categorical, multivariate Gaussian, Bernoulli, Laplace and Wishart distributions. §5 defines Hölder statistical centroids and presents a variational $k$-means clustering technique: We show experimentally that using Hölder divergences improve over the Cauchy-Schwarz divergence. Finally, 6 concludes this work and hints at further perspectives from the viewpoint of statistical estimation and manifold learning. In Appendix A, we recall the proof of the ordinary and reverse Hölder’s inequalities.

2 Hölder pseudo-divergence: Definition and properties

Hölder’s inequality (see [18] and Appendix A for a proof) states for positive real-valued functions333In a more general form, Hölder’s inequality holds for any real and complex valued functions. In this work, we only focus on real positive functions that are densities of positive measures. $p(x)$ and $q(x)$ defined on the support $\mathcal{X}$ that:

[TABLE]

where exponents $\alpha$ and $\beta$ satisfy $\alpha\beta>0$ as well as the exponent conjugacy condition: $\frac{1}{\alpha}+\frac{1}{\beta}=1$. We also write $\beta=\bar{\alpha}=\frac{\alpha}{\alpha-1}$ meaning that $\alpha$ and $\beta$ are conjugate Hölder exponents. We check that $\alpha>1$ and $\beta>1$. Hölder inequality holds even if the lhs is infinite (meaning that the integral diverges) since the rhs is also infinite in that case.

The reverse Hölder inequality holds for conjugate exponents $\frac{1}{\alpha}+\frac{1}{\beta}=1$ with $\alpha\beta<0$ (then $0<\alpha<1$ and $\beta<0$, or $\alpha<0$ and $0<\beta<1$):

[TABLE]

Both Hölder’s inequality and the reverse Hölder inequality turn tight when $p(x)^{\alpha}\propto{}q(x)^{\beta}$ (see proof in Appendix A).

2.1 Definition

Let $(\mathcal{X},\mathcal{F},\mu)$ be a measurable space where $\mu$ is the Lebesgue measure, and let $L^{\gamma}(\mathcal{X},\mu)$ denote the Lebesgue space of functions that have their $\gamma$-th power of absolute value Lebesgue integrable, for any $\gamma>0$ (when $\gamma\geq 1$, $L^{\gamma}(\mathcal{X},\mu)$ is a Banach space). We define the following pseudo-divergence:

Definition 1** (Hölder statistical pseudo-divergence, HPD).**

For conjugate exponents $\alpha$ and $\beta$ with $\alpha\beta>0$, the Hölder Pseudo-Divergence (HPD) between two densities $p(x)\in L^{\alpha}(\mathcal{X},\mu)$ and $q(x)\in L^{\beta}(\mathcal{X},\mu)$ of positive measures absolutely continuous with respect to (wrt.) $\mu$ is defined by the following log-ratio gap:

[TABLE]

When $0<\alpha<1$ and $\beta=\bar{\alpha}=\frac{\alpha}{\alpha-1}<0$, or $\alpha<0$ and $0<\beta<1$, the reverse HPD is defined by:

[TABLE]

By Hölder’s inequality and the reverse Hölder inequality, $D^{\mathtt{H}}_{\alpha}(p(x):q(x))\geq 0$ with $D^{\mathtt{H}}_{\alpha}(p(x):q(x))=0$ iff $p(x)^{\alpha}\propto{q(x)}^{\beta}$ or equivalently $q(x)\propto{p(x)}^{\alpha/\beta}=p(x)^{\alpha-1}$. When $\alpha>1$, $x^{\alpha-1}$ is monotonically increasing, and $D^{\mathtt{H}}_{\alpha}$ is indeed a pseudo-divergence. However, the reverse HPD is not a pseudo-divergence because $x^{\alpha-1}$ will be monotonically decreasing if $\alpha<0$ or $0<\alpha<1$. Therefore we only consider HPD with $\alpha>1$ in the remainder, and leave here the notion of reverse Hölder divergence.

When $\alpha=\beta=2$, the HPD becomes the Cauchy-Schwarz divergence ${\mathrm{CS}}$ [16]:

[TABLE]

which has been proved useful to get closed-form divergence formulas between mixtures of exponential families with conic or affine natural parameter spaces [21].

The Cauchy-Schwarz divergence is proper for probability densities since the Cauchy-Schwarz inequality becomes an equality iff $q(x)=\lambda p(x)^{\alpha-1}=\lambda p(x)$ implies that $\lambda=\int_{\mathcal{X}}\lambda p(x)\mathrm{d}x=\int_{\mathcal{X}}q(x)\mathrm{d}x=1$. It is however not proper for positive densities.

Fact 1** (CS is only proper for probability densities).**

The Cauchy-Schwarz divergence ${\mathrm{CS}}(p(x):q(x))$ is proper for square-integrable probability densities $p(x),q(x)\in L^{2}(\mathcal{X},\mu)$ but not proper for positive square-integrable densities.

2.2 Properness and improperness

In the general case, when $\alpha\neq 2$, the divergence $D^{\mathtt{H}}_{\alpha}$ is not even proper for normalized (probability) densities, not to mention general unnormalized (positive) densities. Indeed, when $p(x)=q(x)$, we have:

[TABLE]

Let us consider the general case. For unnormalized positive distributions $\tilde{p}(x)$ and $\tilde{q}(x)$ (the tilde notation stems from the notation of homogeneous coordinates in projective geometry), the inequality becomes an equality when: $\tilde{p}(x)^{\alpha}\propto\tilde{q}(x)^{\beta}$, i.e., $p(x)^{\alpha}\propto{q(x)}^{\beta}$, or $q(x)\propto{p(x)}^{\alpha/{\bar{\alpha}}}=p(x)^{\alpha-1}$. We can check that $D^{\mathtt{H}}_{\alpha}(p(x):\lambda p(x)^{\alpha-1})=0$ for any $\lambda>0$:

[TABLE]

since $(\alpha-1)\beta=(\alpha-1){\bar{\alpha}}=(\alpha-1)\frac{\alpha}{\alpha-1}=\alpha$.

For $\alpha=2$, we find indeed that $D^{\mathtt{H}}_{2}(p(x):\lambda p(x))={\mathrm{CS}}(p(x):p(x))=0$ for any $\lambda\neq 0$.

Fact 2** (HPD is improper).**

The Hölder pseudo-divergences are improper statistical distances.

2.3 Reference duality

In general, Hölder divergences are asymmetric when $\alpha\neq\beta$ ($\not=2$) but enjoy the following reference duality [39]:

[TABLE]

Fact 3** (Reference duality HPD).**

The Hölder pseudo-divergences satisfy the reference duality $\alpha\leftrightarrow\beta=\frac{\alpha}{\alpha-1}$: $D^{\mathtt{H}}_{\alpha}(p(x):q(x))=D^{\mathtt{H}}_{\beta}(q(x):p(x))=D^{\mathtt{H}}_{\frac{\alpha}{\alpha-1}}(q(x):p(x))$.

An arithmetic symmetrization of the HPD yields a symmetric HPD $S^{\mathtt{H}}_{\alpha}$, given by:

[TABLE]

2.4 HPD is a projective divergence

In the above definition, densities $p(x)$ and $q(x)$ can either be positive or normalized probability distributions. Let $\tilde{p}(x)$ and $\tilde{q}(x)$ denote positive (not necessarily normalized) measures, and $w(\tilde{p})=\int_{\mathcal{X}}\tilde{p}(x)\mathrm{d}x$ the overall mass so that $p(x)=\frac{\tilde{p}(x)}{w(\tilde{p})}$ is the corresponding normalized probability measure. Then we check that HPD is a projective divergence [13] since:

[TABLE]

or in general:

[TABLE]

for all prescribed constants $\lambda,\lambda^{\prime}>0$. Projective divergences may also be called “angular divergences” or “cosine divergences” since they do not depend on the total mass of the measure densities.

Fact 4** (HPD is projective).**

The Hölder pseudo-divergences are projective distances.

2.5 Escort distributions and skew Bhattacharyya divergences

Let us define with respect to the probability measures $p(x)\in L^{\frac{1}{\alpha}}(\mathcal{X},\mu)$ and $q(x)\in L^{\frac{1}{\beta}}(\mathcal{X},\mu)$ the following escort probability distributions [2]:

[TABLE]

and

[TABLE]

Since HPD is a projective divergence, we compute with respect to the conjugate exponents $\alpha$ and $\beta$ the Hölder Escort Divergence (HED):

[TABLE]

which turns out to be the familiar skew Bhattacharyya divergence $B_{1/\alpha}(p(x):q(x))$, see [22].

Fact 5** (HED as a skew Bhattacharyya divergence).**

The Hölder escort divergence amounts to a skew Bhattacharyya divergence: $D^{\mathrm{HE}}_{\alpha}(p(x):q(x))=B_{1/\alpha}(p(x):q(x))$ for any $\alpha>0$.

In particular, the Cauchy-Schwarz escort divergence ${\mathrm{CS}}^{\mathrm{HE}}(p(x):q(x))$ amounts to the Bhattacharyya distance [6] $B(p(x):q(x))=-\log\sqrt{\int_{\mathcal{X}}p(x)q(x)\mathrm{d}x}$:

[TABLE]

Observe that the Cauchy-Schwarz escort distributions are the square root density representations [36] of distributions.

3 Proper Hölder divergence

3.1 Definition

Let $p(x)$ and $q(x)$ be positive measures in $L^{\gamma}(\mathcal{X},\mu)$ for a prescribed scalar value $\gamma>0$. Plugging the positive measures $p(x)^{\gamma/\alpha}$ and $q(x)^{\gamma/\beta}$ into the definition of HPD $D^{\mathtt{H}}_{\alpha}$, we get the following definition:

Definition 2** (Proper Hölder Divergence, HD).**

For conjugate exponents $\alpha,\beta>0$ and $\gamma>0$, the proper Hölder divergence between two densities $p(x)$ and $q(x)$ is defined by:

[TABLE]

By definition, $D^{\mathtt{H}}_{\alpha,\gamma}(p:q)$ is a two-parameter family of dissimilarity statistical measures. Following Hölder’s inequality, we can check that $D^{\mathtt{H}}_{\alpha,\gamma}(p(x):q(x))\geq 0$ and $D^{\mathtt{H}}_{\alpha,\gamma}(p(x):q(x))=0$ iff $p(x)^{\gamma}\propto{q}(x)^{\gamma}$, i.e. $p(x)\propto{q}(x)$ (see Appendix A). If $p(x)$ and $q(x)$ belong to the statistical probability manifold, then $D^{\mathtt{H}}_{\alpha,\gamma}(p(x):q(x))=0$ iff $p(x)=q(x)$ almost everywhere. This says that HD is a proper divergence for probability measures, and it becomes a pseudo-divergence for positive measures. Note that we have abused the notation $D^{\mathtt{H}}$ to denote both the Hölder pseudo-divergence (with one subscript) and the Hölder divergence (with two subscripts).

Similar to HPD, HD is asymmetric when $\alpha\neq\beta$ with the following reference duality:

[TABLE]

HD can be symmetrized as:

[TABLE]

Furthermore, one can easily check that HD is a projective divergence.

For conjugate exponents $\alpha,\beta>0$ and $\gamma>0$, we rewrite the definition of HD as:

[TABLE]

Therefore HD can be reinterpreted as the skew Bhattacharyya divergence [22] between the escort distributions. In particular, when $\gamma=1$, we get:

[TABLE]

Fact 6**.**

The two-parametric family of statistical Hölder divergence $D^{\mathtt{H}}_{\alpha,\gamma}$ passes through the one-parametric family of skew Bhattacharyya divergences when $\gamma=1$.

3.2 Special case: The Cauchy-Schwarz divergence

We consider the intersection of the uni-parametric class of Hölder pseudo-divergences (HPD) with the bi-parametric class of proper Hölder divergences (HD): That is, the class of divergences which belong to both HPD and HD. Then we must have $\gamma/\alpha=\gamma/\beta=1$. Since $1/\alpha+1/\beta=1$, we get $\alpha=\beta=\gamma=2$. Therefore the Cauchy-Schwarz (CS) divergence is the unique divergence belonging to both HPD and HD classes:

[TABLE]

In fact, the CS divergence is the intersection of the four classes HPD, symmetric HPD, HD, and symmetric HD. Figure 1 displays a diagram of those divergence classes with their inclusion relationships.

As stated earlier, notice that the Cauchy-Schwarz inequality:

[TABLE]

is not proper as it is an equality when $p(x)$ and $q(x)$ are linearly dependent (i.e., $p(x)=\lambda q(x)$ for $\lambda>0$). The arguments of the CS divergence are square-integrable real-valued density functions $p(x)$ and $q(x)$. Thus the Cauchy-Schwarz divergence is not proper for positive measures but is proper for normalized probability distributions since $\int p(x)\mathrm{d}x=\int\lambda q(x)\mathrm{d}x=1$ implies that $\lambda=1$.

3.3 Limit cases of Hölder divergences and statistical estimation

Let us define the inner product of unnormalized densities as:

[TABLE]

(for $L^{2}(\mathcal{X},\mu)$ integrable functions), and define the $L_{\alpha}$ norm of densities as $\|\tilde{p}(x)\|_{\alpha}=(\int_{\mathcal{X}}\tilde{p}(x)^{\alpha}\mathrm{d}x)^{1/\alpha}$ for $\alpha\geq 1$. Then the CS divergence can be concisely written as:

[TABLE]

and the Hölder pseudo-divergence writes as:

[TABLE]

When $\alpha\rightarrow 1^{+}$, we have ${\bar{\alpha}}=\alpha/(\alpha-1)\rightarrow+\infty$. Then it comes that:

[TABLE]

When $\alpha\to+\infty$ and ${\bar{\alpha}}\to 1^{+}$, we have:

[TABLE]

Now consider a pair of probability densities $p(x)$ and $q(x)$. We have:

[TABLE]

In an estimation scenario, $p(x)$ is fixed and $q(x\,|\,\theta)=q_{\theta}(x)$ is free along a parametric manifold $\mathcal{M}$, then minimizing Hölder divergence reduces to:

[TABLE]

Therefore when $\theta$ varies from 1 to $+\infty$, only the regularizer in the minimization problem changes. In any case, Hölder divergence always has the term $-\log{\langle p(x),q(x)\rangle}$, which shares a similar form with the Bhattacharyya distance [6]:

[TABLE]

HPD between $\tilde{p}(x)$ and $\tilde{q}(x)$ is also closely related to their cosine similarity $\frac{{\langle\tilde{p}(x),\tilde{q}(x)\rangle}}{\|\tilde{p}(x)\|_{2}\|\tilde{q}(x)\|_{2}}$. When $\alpha=2$, HD is exactly the cosine similarity after a non-linear transformation.

4 Closed-form expressions of HPD and HD for conic and affine exponential families

We report closed-form formulas for the HPD and HD between two distributions belonging to the same exponential family provided that the natural parameter space is a cone or affine. A cone $\Omega$ is a convex domain such that for $P,Q\in\Omega$ and any $\lambda>0$, we have $P+\lambda Q\in\Omega$. For example, the set of positive measures absolutely continuous with a base measure $\mu$ is a cone. Recall that an exponential family [23] has a density function $p(x;\theta)$ that we be written canonically as:

[TABLE]

In this work, we consider the auxiliary carrier measure term $k(x)=0$. The base measure is either the Lebesgue measure $\mu$ or the counting measure $\mu_{C}$. A Conic or Affine Exponential Family (CAEF) is an exponential family with the natural parameter space $\Theta$ a cone or affine. The log-normalizer $F(\theta)$ is a strictly convex function also called cumulant generating function [2].

Lemma 1** (HPD and HD for CAEFs).**

For distributions $p(x;\theta_{p})$ and $p(x;\theta_{q})$ belonging to the same exponential family with conic or affine natural parameter space [21], both the HPD and HD are available in closed-form:

[TABLE]

Proof.

Consider $k(x)=0$ and a conic or affine natural space $\Theta$ (see [21]), then for all $a,b>0$, we have:

[TABLE]

since $a\theta_{p}\in\Theta$. Indeed, we have:

[TABLE]

Similarly, we have for all ${a},{b}>0$ (details omitted),

[TABLE]

since $a\theta_{p}+b\theta_{q}\in\Theta$. Therefore, we get:

[TABLE]

When $1>\alpha>0$, we have $\beta=\frac{\alpha}{\alpha-1}<0$. To get similar results for the reverse Hölder divergence, we need the natural parameter space to be affine (eg., isotropic Gaussians or multinomials, see [27]). ∎

In particular, if $p(x)$ and $q(x)$ belong to the same exponential family so that $p(x)=\exp({\langle\theta_{p},t(x)\rangle}-F(\theta_{p}))$ and $q(x)=\exp({\langle\theta_{q},t(x)\rangle}-F(\theta_{q}))$, one can easily check that $D^{\mathtt{H}}_{\alpha}(p(x;\theta_{p}):p(x;\theta_{q}))=0$ iff $\theta_{q}=(\alpha-1)\theta_{p}$. For HD, we can check $D^{\mathtt{H}}_{\alpha,\gamma}(p(x):p(x))=0$ is proper since $\frac{1}{\alpha}+\frac{1}{\beta}=1$.

The following result is straightforward from Lemma 1.

Lemma 2** (Symmetric HPD and HD for CAEFs).**

For distributions $p(x;\theta_{p})$ and $p(x;\theta_{q})$ belonging to the same exponential family with conic or affine natural parameter space [21], the symmetric HPD and HD are available in closed-form:

[TABLE]

Remark 1**.**

By reference duality,

[TABLE]

Note that the Hölder score-induced divergence [19] does not admit in general closed-form formulas for exponential families since it relies on a function $\phi(\cdot)$ (see Definition 4 of [19]).

Note that CAEF convex log-normalizers satisfy:

[TABLE]

A necessary condition is that $F(\lambda\theta)\geq\lambda F(\theta)$ for $\lambda>0$ (take $\theta_{p}=\theta$, $\theta_{q}=0$ and $F(0)=0$ in the above equality).

The escort distribution for an exponential family is given by:

[TABLE]

The Hölder equality holds when $p(x)^{\alpha}\propto q(x)^{\beta}$ or $p(x)^{\alpha}q(x)^{-\beta}\propto 1$. For exponential families, this condition is satisfied when $\alpha\theta_{p}-\beta\theta_{q}\in\Theta$. That is, we need to have:

[TABLE]

Thus we may choose small enough $\alpha=1+\epsilon>1$ so that the condition is not satisfied for fixed $\theta_{p}$ and $\theta_{q}$ for many exponential distributions. Since multinomials have affine natural space [27], this condition is always met, but not for non-affine natural parameter spaces like normal distributions.

Notice the following fact:

Fact 7** (Density of a CAEF in $L^{\gamma}(\mathcal{X},\mu)$).**

The density of exponential families with conic or affine natural parameter space belongs to $L^{\gamma}(\mathcal{X},\mu)$ for any $\gamma>0$.

Proof.

We have $\int_{\mathcal{X}}(\exp({\langle\theta,t(x)\rangle}-F(\theta)))^{\gamma}\mathrm{d}\mu(x)=e^{F(\gamma\theta)-\gamma F(\theta)}<\infty$ for any $\gamma>0$ provided that $\gamma\theta$ belongs to the natural parameter space. When $\Theta$ is a cone or affine, the condition is satisfied. ∎

Let $\tilde{p}(x;\theta)=\exp\left({\langle t(x),\theta)\rangle}\right)$ denote the unnormalized positive exponential family density and $p(x;\theta)=\frac{\tilde{p}(x;\theta)}{Z(\theta)}$ the normalized density with $Z(\theta)=\exp(F(\theta))$ the partition function. Although HD is a projective divergence since we have $D^{\mathtt{H}}_{\alpha}(p(x;\theta_{1}):p(x;\theta_{2}))=D^{\mathtt{H}}_{\alpha}(\tilde{p}(x;\theta_{1}):\tilde{p}(x;\theta_{2}))$, observe that the HD value depends on the log-normalizer $F(\theta)$ (since the HD is an integral on the support, see also [29] for a similar argument with the $\gamma$-divergence [13]).

In practice, even when the log-normalizer is computationally intractable, we may still estimate the HD by Monte-Carlo techniques: Indeed, we can sample a distribution $\tilde{p}(x)$ either by rejection sampling [29] or by the Markov Chain Monte-Carlo (MCMC) Metropolis-Hasting technique: It just requires to be able to sample a proposal distribution that has the same support.

We shall now instantiate the HPD and HD formulas for several exponential families with conic or affine natural parameter spaces.

4.1 Case study: Categorical distributions

Let $p=(p_{0},\cdots,p_{m})$ and $q=(q_{0},\cdots,q_{m})$ be two categorical distributions in the $m$-dimensional probability simplex $\Delta^{m}$. We rewrite $p$ in the canonical form of exponential families [23] as:

[TABLE]

with the redundant parameter:

[TABLE]

From Eq. 55, the convex cumulant generating function has the form $F(\theta)=\log\left(1+\sum_{i=1}^{m}\exp(\theta_{p})_{i}\right)$. The inverse transformation from $p$ to $\theta$ is therefore given by:

[TABLE]

The natural parameter space $\Theta$ is affine (hence conic), and by applying Lemma 1, we get the following closed-form formula:

[TABLE]

To get some intuitions, Fig. 2 shows the Hölder divergence from any categorical distribution $(p_{0},p_{1},p_{2})$ to a given reference distribution $p_{r}$ in the 2D probability simplex $\Delta^{2}$. A main observation is that the Kullback-Leibler (KL) divergence exhibits a barrier near the boundary $\partial\Delta^{2}$ with large values. This is not the case for Hölder divergences: $D^{\mathtt{H}}_{\alpha}(p:p_{r})$ does not have a sharp increase near the boundary (although it still penalizes the corners of $\Delta^{2}$). For example, let $p=(0,1/2,1/2)$, $p_{r}=(1/3,1/3,1/3)$, then $\mathrm{KL}(p_{r}:p)\to\infty$ but $D^{\mathtt{H}}_{2}(p_{r}:p)=\sqrt{2/3}$. Another observation is that the minimum $D(p:p_{r})$ can be reached at some point $p\neq{p_{r}}$ (see for example $D^{\mathtt{H}}_{4}(p:p_{r})$ in Fig. 2(b), the bluest area corresponding to the minimum of $D(p:p_{r})$ is not in the same location as the reference point).

Consider a HPD ball of center $c$ and prescribed radius $r$ wrt the HPD. Since $p(x)^{\alpha-1}$ for $\alpha\not=2$ does not belong to the probability manifold but to the positive measure manifold and since the distance is projective, we deduce that the displaced ball center $c^{\prime}$ of a ball $c$ lying the probability manifold can be computed as the intersection of the ray $0p^{\alpha-1}$ with the probability manifold. For the discrete probability simplex $\Delta$, since we have $\lambda\sum_{x\in\mathcal{X}}p(x)^{\alpha-1}=1$, we deduce that the displaced ball center is at:

[TABLE]

This center is displayed as “$\bullet$” in Figure 2.

In general, the HPD bisector [26] between two distributions belonging to the same CAEF is defined by:

[TABLE]

4.2 Case study: Bernoulli distribution

Bernoulli distribution is just a special case of the category distribution when the number of categories is $2$ (i.e., $m=1$). To be consistent with the previous section, we rewrite a Bernoulli distribution $p=(p_{0},p_{1})$ in the canonical form:

[TABLE]

so that

[TABLE]

Then the cumulant generating function becomes $F(\theta_{p})=\log\left(1+\exp(\theta_{p})\right)$. By Lemma 1,

[TABLE]

4.3 Case study: MultiVariate Normal distributions (MVNs)

Let us now instantiate the formulas for multivariate normals (Gaussian distributions). We have the log-normalizer $F(\theta)$ expressed using the usual parameters as [25]:

[TABLE]

Since

[TABLE]

It follows that:

[TABLE]

Therefore, we have:

[TABLE]

We thus get the following closed-form formula for $p~{}\sim N(\mu_{p},\Sigma_{p})$ and $q~{}\sim N(\mu_{q},\Sigma_{q})$:

[TABLE]

Figure 3 shows HPD and HD for univariate Gaussian distributions as compared to the KL divergence.

4.4 Case study: Zero-centered Laplace distribution

The zero-centered Laplace distribution is defined on the support $(-\infty,\infty)$ with the pdf:

[TABLE]

We have $\theta=-\frac{1}{\sigma}$, $F(\theta)=\log(-\frac{2}{\theta})$. Therefore, it comes that:

[TABLE]

In this special case, $D^{\mathtt{H}}_{\alpha,\gamma}(p(x):q(x))$ does not vary with $\gamma$.

4.5 Case study: Wishart distribution

The Wishart distribution is defined on the $d\times{d}$ positive definite cone with the density:

[TABLE]

where $n>d-1$ is the degree of freedom, and $S\succ 0$ is a scale matrix. We rewrite it into the canonical form:

[TABLE]

We can see that $\theta=(\theta^{1},\theta^{2})$, $\theta^{1}=-\frac{1}{2}S^{-1}$, $\theta^{2}=\frac{n-d-1}{2}$, and

[TABLE]

The resulting $D^{\mathtt{H}}_{\alpha}(p(x):q(x))$ and $D^{\mathtt{H}}_{\alpha,\gamma}(p(x):q(x))$ are straightforward from the above expression of $F(\theta)$ and Lemma 1. We will omit these tedious expressions for brevity.

4.6 Approximating Hölder projective divergences for statistical mixtures

Given two finite mixture models $m(x)=\sum_{i=1}^{k}w_{i}p_{i}(x)$ and $m^{\prime}(x)=\sum_{j=1}^{k^{\prime}}w_{j}^{\prime}p_{j}^{\prime}(x)$, we derive analytic bounds of their Hölder divergences. When approximation is only needed, one may compute Hölder divergences based on Monte-Carlo stochastic sampling.

Let us assume that all mixture components are in an exponential family [23] so that $p_{i}(x)=p(x;\theta_{i})=\exp({\langle\theta_{i},t(x)\rangle}-F(\theta_{i}))$ and $p_{j}^{\prime}(x)=p(x;\theta_{j}^{\prime})=\exp({\langle\theta_{j}^{\prime},t(x)\rangle}-F(\theta_{j}^{\prime}))$ are densities (wrt the Lebesgue measure $\mu$).

We rewrite the Hölder divergence into the form:

[TABLE]

To compute the first term, we observe that a product of mixtures is also a mixture:

[TABLE]

which can be computed in $O(kk^{\prime})$ time.

The second and third terms in Eq. 78 are not straightforward to calculate and shall be bounded. Based on computational geometry, we adopt the log-sum-exp bounding technique of [30] and divide the support $\mathcal{X}$ into $L$ pieces of elementary intervals $\mathcal{X}=\biguplus{}_{l=1}^{L}I_{l}$. In each interval $I_{l}$, the indices:

[TABLE]

representing the unique dominating component and the dominated component. Then we bound as follows:

[TABLE]

All terms on the lhs and rhs of Eq. 81 can be computed exactly by noticing that:

[TABLE]

When $\alpha\theta\in\Theta$ where $\Theta$ denotes the natural parameter space, the integral $\int_{I}p(x;\alpha\theta_{i})\mathrm{d}x$ converges, see [30] for further details.

Then the bounds of $\int_{\mathcal{X}}m(x)^{\alpha}\mathrm{d}x$ can be obtained by summing the bounds in Eq. 81 over all elementary intervals. Thus $D^{\mathtt{H}}_{\alpha,\beta}(m(x):m^{\prime}(x))$ can be both lower and upper bounded.

5 Hölder centroids and center-based clustering

We study the application of HPD and HD for clustering distributions [31], specially clustering Gaussian distributions [35, 12, 24], which have been used in sound processing [35], sensor network [12], statistical debugging [12], and quadratic invariants of switched systems [1], etc. Other potential applications of HD may include nonnegative matrix factorization [37], and clustering von Mises-Fisher [3, 15] (log-normalizer expressed using Bessel functions).

5.1 Hölder centroids

We study center-based clustering of a finite set of distributions: Given a list of distributions belonging to the same conic exponential family with natural parameters $\{\theta_{1},\cdots,\theta_{n}\}$ and their associated positive weights $\{w_{1},\cdots,w_{n}\}$ with $\sum_{i=1}^{n}w_{i}=1$, consider their centroids based on HPD and HD as follows:

[TABLE]

By abuse of notation, $C$ denotes both the HPD centroid and HD centroid. When the context is clear, the parameters in parentheses can be omitted so that these centroids are simply denoted as $C_{\alpha}$ and $C_{\alpha,\gamma}$. Both of them are defined as the right-sided centroids. The corresponding left-handed centroids are obtained according to reference duality, i.e.,

[TABLE]

By Lemma 1, these centroids can be obtained for distributions belonging to the same exponential family as follows:

[TABLE]

Let $\gamma=\alpha$, we get:

[TABLE]

meaning that the HPD centroid is just a special case of HD centroid up to a scaling transformation in the natural parameters space. Let $\gamma=\beta$, we get:

[TABLE]

Let us consider the general HD centroid $C_{\alpha,\gamma}$. Since $F$ is convex, the minimization energy is the sum of a convex function $\frac{1}{\beta}F(\gamma C)$ with a concave function $-\sum_{i=1}^{n}w_{i}F\left(\frac{\gamma}{\alpha}\theta_{i}+\frac{\gamma}{\beta}C\right)$. We can therefore use the concave-convex procedure (CCCP) [22] that optimizes Difference of Convex Programs (DCPs): We start with $C_{\alpha,\gamma}^{0}=\sum_{i=1}^{n}w_{i}\theta_{i}$ (the barycenter, belonging to $\Theta$) and then update:

[TABLE]

for $t=0,1,\cdots$ until convergence. This can be done by noting that $\eta=\nabla F(\theta)$ are the dual parameters that are also known as the expectation parameters (or moment parameters). Therefore $\nabla{F}$ and $(\nabla{F})^{-1}$ can be computed through Legendre transformations between the natural parameter space and the dual parameter space.

This iterative optimization is guaranteed to converge to a local minimum, with a main advantage of bypassing the learning rate parameter of gradient descent algorithms. Since $F$ is strictly convex, $\nabla F$ is monotonous, and the rhs expression can be interpreted as a multi-dimensional quasi-arithmetic mean. In fact, it is a barycenter on non-normalized weights scaled by $\beta={\bar{\alpha}}$.

For exponential families, the symmetric HPD centroid is:

[TABLE]

In this case, CCCP update rule is not in closed form because we cannot easily inverse the sum of gradients (but when $\alpha=\beta$, the two terms collapse so CS centroid can be calculated using CCCP). Nevertheless, we can implement the reciprocal operation numerically. Interestingly, the symmetric HD centroid can be solved by CCCP! It amounts to solve:

[TABLE]

One can apply CCCP to iteratively update the centroids based on:

[TABLE]

Notice the similarity with the updating procedure of $C_{\alpha,\gamma}^{t}$.

Once the centroid, say $O_{\alpha,\gamma}$, has been computed, we calculate the associated Hölder information:

[TABLE]

The Hölder information generalizes the notion of variance and Bregman information [4] to the case of Hölder distances.

5.2 Clustering based on symmetric Hölder divergences

Given a set of fixed densities $\{p_{1},\cdots,p_{n}\}$, we can perform variational $k$-means [28] with respect to the Hölder divergence to minimize the cost function:

[TABLE]

where $O_{1},\cdots,O_{L}$ are the cluster centers, and $l_{i}\in\{1,\cdots,L\}$ is the cluster label of $p_{i}$. The algorithm is given by Algorithm 1. Notice that one does not need to wait for the CCCP iterations to converge. It only has to improve the cost function $E$ before updating the assignment. We have implemented the algorithm based on the symmetric HD. One can easily modify it based on HPD and other variants.

We made a toy dataset generator which can randomly generate $n$ 2D Gaussians which have a underlying structure of two clusters. In the first cluster, the mean of each Gaussian $G(\mu,\Sigma)$ has the prior distribution $\mu\sim G((-2,0),I)$; the covariance matrix is obtained by first generating $\sigma_{1}\sim\Gamma(7,0.01)$, $\sigma_{2}\sim\Gamma(7,0.003)$, where $\Gamma$ means a gamma distribution with prescribed shape and scale, then rotating the covariance matrix $\mathrm{diag}(\sigma_{1},\sigma_{2})$ so that the resulting Gaussian has a “radial direction” with respect to the center $(-2,0)$. The second cluster is similar to the first cluster with the only difference being that $\mu\sim G((2,0),I)$ is centered around $(-2,0)$. See Fig. 4 for an intuitive presentation of the toy dataset.

To reduce the number of parameters that has to be tuned, we only investigate the case $\alpha=\gamma$. If we choose $\alpha=\gamma=2$, then $S^{\mathtt{H}}_{\alpha,\gamma}$ becomes the CS divergence and Algorithm 1 reduces to traditional CS clustering. From Fig. 4 we can observe that the clustering result does vary with the settings of $\alpha$ and $\gamma$. We performed clustering experiments on two different settings of the sample size. Table 1 shows the clustering accuracy measured by the percentage of “correctly clustered” Gaussians, i.e., the output label by clustering algorithms that coincides with the true label corresponding to the data generating process. We see that the symmetric Hölder divergence can give better clustering results as compared to CS clustering. This hints that one could consider the general Hölder divergence to replace CS in similar clustering applications [16, 33]. Although one faces the problem of tuning the parameter $\alpha$ and $\gamma$, Hölder divergences can potentially give better results. This is expected because CS is just one particular case of the class of Hölder divergences.

6 Conclusion and perspectives

We introduced the notion of pseudo-divergences that generalizes the concept of divergences in information geometry [2] that are smooth non-metric statistical distances that are not required to obey the law of the indiscernibles. Pseudo-divergences can be built from inequalities by considering the inequality difference gap or its log-ratio gap. We then defined two classes of statistical measures based on Hölder’s ordinary and reverse inequalities: The singly parametric family of Hölder pseudo-divergences and the doubly parametric family of Hölder divergences. By construction, the Hölder divergences are proper divergences between probability densities. Both statistical Hölder distance families are projective divergences that do not require distributions to be normalized, and admit closed-form expressions when considering exponential families with conic or affine natural parameter space (like multinomials or multivariate normals). Those two families of distances can be symmetrized and intersect at the unique Cauchy-Schwarz divergence. Since the Cauchy-Schwarz divergence is often used in distribution clustering applications [16], we carried out preliminary experiments demonstrating experimentally that thesymmetrized Hölder divergences improved over the Cauchy-Schwarz divergence for a toy dataset of Gaussians. We briefly touched upon the use of these novel divergences in statistical estimation theory. These projective Hölder (pseudo-)divergences are different from the recently introduced compositive scored-induced Hölder divergences [20, 19] that are not projective divergences and do not admit closed-form expressions for exponential families in general.

We elicited the special role of escort distributions [2] for Hölder divergences in our framework: Escort distributions transform distributions to allow one:

• •

To reveal that Hölder pseudo-divergences on escort distributions amount to skew Bhattacharyya divergences [22],

• •

To transform the improper Hölder pseudo-divergences into proper Hölder divergences, and vice versa.

Let us conclude with a perspective note on pseudo-divergences, statistical estimators, and manifold learning. Proper divergences have been widely used in statistical estimators to build families of estimators [32, 5]. Similarly, given a prescribed density $p_{0}(x)$, a pseudo-divergence yields a corresponding estimator by minimizing $D(p_{0}(x):q(x))$ with respect to $q(x)$. However in this case the resulting $q(x)$ is potentially biased and is not guaranteed to recover the optimal input $p_{0}(x)$. Furthermore, the minimizer of $D(p_{0}(x):q(x))$ may not be unique, i.e., there could be more than one probability density $q(x)$ yielding $D(p_{0}(x):q(x))=0$.

How can pseudo-divergences be useful? We have the following two simple arguments:

• •

In an estimation scenario, we can usually pre-compute $p_{1}(x)\neq{p_{0}}(x)$ according to $D(p_{1}(x):p_{0}(x))=0$. Then the estimation $q(x)=\arg\min_{q}D(p_{1}(x):q(x))$ will automatically target at $p_{0}(x)$. We call this technique pre-aim.

For example, given positive measure $p(x)$, we first find $p_{0}(x)$ to satisfy $D^{\mathtt{H}}_{\alpha}(p_{0}(x):p(x))=0$. We have $p_{0}(x)=p(x)^{\frac{1}{\alpha-1}}$ that satisfies this condition. Then a proper divergence between $p(x)$ and $q(x)$ can be obtained by aiming $q(x)$ towards $p_{0}(x)$. For conjugate exponents $\alpha$ and $\beta$,

[TABLE]

This means that the pre-aim technique of HPD is equivalent to HD $D^{\mathtt{H}}_{\alpha,\gamma}$ when we set $\gamma=\beta$.

As an alternative implementation of pre-aim, since $D^{\mathtt{H}}_{\alpha}(p(x):p^{\alpha-1}(x))=0$, a proper divergence between $p(x)$ and $q(x)$ can be constructed by measuring:

[TABLE]

turning out again to belong to the class of HD.

In practice, HD as a two-parameter family is less used than HPD with pre-aim because of the difficulty in choosing the parameter $\gamma$, and because that HD has a slightly more complicated expression. The family of HD connecting CS divergence with skew Bhattacharyya divergence [22] is nevertheless of theoretical importance.

• •

In manifold learning [17, 10, 38], it is an essential topic to align two category distributions $p_{0}(x)$ and $q(x)$ corresponding respectively to the input and output [38], both for learning and for performance evaluation. In this case, the dimensionality of the statistical manifold that encompasses $p_{0}(x)$ and $q(x)$ is so high that to preserve monotonically $p_{0}(x)$ in the resulting $q(x)$ is already a difficult non-linear optimization and could be sufficient for the application, while preserving perfectly the input $p_{0}(x)$ is not so meaningful because of the input noise. It is then much easier to define pseudo-divergences using inequalities which are not necessarily to be proper with potentially more choices. On the other hand, projective divergences including Hölder divergences introduced in this work are more meaningful in manifold learning than KL divergence (which is widely used) because they give scale invariance of the probability densities, meaning that one can define positive similarities then directly align these similarities, which is guaranteed to be equivalent to align the corresponding distributions. This could potentially give unified perspectives in between the two approaches of similarity based manifold learning [10] and the probabilistic approach [17].

We expect that these two novel parametric Hölder classes of statistical divergences and pseudo-divergences open up new insights and applications in statistics and information sciences. Furthermore, the framework to build divergences or pseudo-divergences from proper or improper biparametric inequalities offers novel classes of divergences to study.

Reproducible source code is available online at:

https://www.lix.polytechnique.fr/~nielsen/HPD/

Appendix A Proof of Hölder ordinary and reverse inequalities

We extend the proof ([7], pp.78) to prove both the (ordinary or forward) Hölder inequality and the reverse Hölder inequality.

Proof.

First, let us observe that $-\log(x)$ is strictly convex on $(0,+\infty)$ since $(-\log(x))^{\prime\prime}=\frac{1}{x^{2}}$. It follows that for $0<a<1$ that:

[TABLE]

where the equality holds iff $x_{1}=x_{2}$.

Conversely, when $a<0$ or $a>1$, we have:

[TABLE]

where the equality holds iff $x_{1}=x_{2}$.

Equivalently, we can write these two inequalities as follows:

[TABLE]

both of them are tight iff $x_{1}=x_{2}$.

Let $P$ and $Q$ be positive measures with Radon-Nikodym densities $p(x)>0$ and $q(x)>0$ be positive densities with respect to the reference Lebesgue measure $\mu$. The densities are strictly greater than [math] on the support $\mathcal{X}$. Plugging:

[TABLE]

into Eq. 103, we get:

[TABLE]

Assume that $p(x)$ in $L_{\alpha}(\mathcal{X},\mu)$ and $q(x)$ in $L_{\beta}(\mathcal{X},\mu)$ so that both $\int_{\mathcal{X}}p(x)^{\alpha}\mathrm{d}x$ and $\int_{\mathcal{X}}q(x)^{\beta}\mathrm{d}x$ converge. Integrate both sides on $\mathcal{X}$ to get:

[TABLE]

The necessary and sufficient condition for equality is that:

[TABLE]

almost everywhere. That is, there exists a positive constant $\lambda>0$ such that:

[TABLE]

∎

The Hölder conjugate exponents $\alpha$ and $\beta$ satisfies $\frac{1}{\alpha}+\frac{1}{\beta}=1$. That is $\beta=\frac{\alpha}{\alpha-1}$. Thus when $\alpha<0$ we necessarily have $\beta>0$, and vice-versa.

We can unify these two straight and reverse Hölder inequalities into a single inequality by considering the sign of $\alpha\beta=\frac{\alpha^{2}}{\alpha-1}$: We get the general Hölder inequality:

[TABLE]

When $\alpha=\beta=2$, Hölder inequality becomes the Cauchy-Schwarz inequality:444Historically, Cauchy stated the discrete sum inequality in 1821 while Schwarz reported the integral form of the inequality in 1888.

[TABLE]

Appendix B Notations

[TABLE]

Contents

1. 1 Introduction: Inequality, proper divergence and improper pseudo-divergence

2. 1.1 Statistical divergences from inequality gaps

3. 1.2 Pseudo-divergences and the axiom of indiscernability

4. 1.3 Prior work and contributions

5. 1.4 Organization

6. 2 Hölder pseudo-divergence: Definition and properties

7. 2.1 Definition

8. 2.2 Properness and improperness

9. 2.3 Reference duality

10. 2.4 HPD is a projective divergence

11. 2.5 Escort distributions and skew Bhattacharyya divergences

12. 3 Proper Hölder divergence

13. 3.1 Definition

14. 3.2 Special case: The Cauchy-Schwarz divergence

15. 3.3 Limit cases of Hölder divergences and statistical estimation

16. 4 Closed-form expressions of HPD and HD for conic and affine exponential families

17. 4.1 Case study: Categorical distributions

18. 4.2 Case study: Bernoulli distribution

19. 4.3 Case study: MultiVariate Normal distributions (MVNs)

20. 4.4 Case study: Zero-centered Laplace distribution

21. 4.5 Case study: Wishart distribution

22. 4.6 Approximating Hölder projective divergences for statistical mixtures

23. 5 Hölder centroids and center-based clustering

24. 5.1 Hölder centroids

25. 5.2 Clustering based on symmetric Hölder divergences

26. 6 Conclusion and perspectives

27. A Proof of Hölder ordinary and reverse inequalities

28. B Notations