Equivalence of Informations Characterizes Bregman Divergences

TL;DR

This paper shows that Bregman divergences are uniquely defined by their ability to equate two measures of information in data sets.

Contribution

The paper proves that Bregman divergences are the only class of divergences that equate Jensen gap and divergence information for arbitrary weighted data sets.

Findings

Bregman divergences uniquely equate Jensen gap and divergence information measures.

This equivalence characterizes the entire class of Bregman divergences.

The result holds for arbitrary weighted sets of data vectors.

Abstract

Bregman divergences form a class of distance-like comparison functions which plays fundamental roles in optimization, statistics, and information theory. One important property of Bregman divergences is that they generate agreement between two useful formulations of information content (in the sense of variability or non-uniformity) in weighted collections of vectors. The first of these is the Jensen gap information, which measures the difference between the mean value of a strictly convex function evaluated on a weighted set of vectors and the value of that function evaluated at the centroid of that collection. The second of these is the divergence information, which measures the mean divergence of the vectors in the collection from their centroid. In this brief note, we prove that the agreement between Jensen gap and divergence informations in fact characterizes the class of Bregman divergences; they are the only divergences that generate this agreement for arbitrary weighted sets of data vectors.

Figures

Full text

1. Introduction

For a convex set $[eqn]$ with relative interior $[eqn]$ and a strictly convex function $[eqn]$ differentiable on $[eqn]$ , the Bregman divergence induced by $[eqn]$ is the function $[eqn]$ defined by

$$[eqn]$$

Two common examples of Bregman divergences are:

• The squared Mahalanobis distance $[eqn]$ , where $[eqn]$ is a positive-definite matrix. The function $[eqn]$ is given by $[eqn]$ . The special case $[eqn]$ gives the squared Euclidean distance. This divergence may be defined on $[eqn]$ .
• The Kullback–Leibler (KL) divergence $[eqn]$ , where $[eqn]$ and $[eqn]$ are probability vectors. Here, $[eqn]$ is the simplex $[eqn]$ . The KL divergence is induced by the negative entropy function $[eqn]$ . This divergence can be extended to general convex subsets of $[eqn]$ with formula $[eqn]$ for $[eqn]$ . When computing the KL divergence, we use the convention $[eqn]$ .

While it is possible to extend the definition of Bregman divergences to Banach spaces [1], in this note we focus on divergences whose domains are convex subsets of $[eqn]$ . In this setting, it is possible to interpret the Bregman divergence as a comparison between the difference $[eqn]$ on the one hand and the linearized approximation of this difference about $[eqn]$ given by $[eqn]$ on the other.

Like metrics, Bregman divergences are positive-definite: $[eqn]$ with equality if and only if $[eqn]$ . Unlike metrics, Bregman divergences are not in general symmetric and do not in general satisfy a triangle inequality, though they do satisfy a “law of cosines” and a generalized Pythagorean theorem [2]. Bregman divergences are locally distance-like in that they induce a Riemannian metric on $[eqn]$ obtained by the small- $[eqn]$ expansion

$$[eqn]$$

where $[eqn]$ is a small perturbation vector and $[eqn]$ is the Hessian of $[eqn]$ at $[eqn]$ . Because $[eqn]$ is strictly convex, $[eqn]$ is positive-definite and defines a Riemannian metric on $[eqn]$ [3]; much work in information geometry [4] pursues the geometry induced by this metric and its connections to statistical inference. Bregman divergences [5] also play fundamental roles in machine learning, optimization, and information theory. They are the unique class of distance-like losses for which iterative, centroid-based clustering algorithms (such as k-means) always reduce the global loss [2,6] Bregman divergences are central in the formulation of mirror-descent methods for convex optimization [7] and have a connection via convex duality to Fenchel-Young loss functions [4,8]. See Reem et al. [9] for a more detailed review of Bregman divergences.

Bregman divergences provide one natural route through which to generalize Shannon information theory, with the differentiable function $[eqn]$ taking on the role of the Shannon entropy. Indeed, generalized entropies play a role in describing the the asymptotic performance of learning algorithms; there exist a number of inequalities relating Bregman divergences to these generalized entropies [10,11]. Multiple characterization theorems exist for many information-theoretic quantities, including entropy [12,13,14], mutual information [15,16], and the Kullback–Leibler divergence [17,18]. This author, however, is aware of only one extant characterization of the class of Bregman divergences, due to Banerjee et al. [6]: Bregman divergences are the unique class of loss functions that render conditional expectations uniquely loss-minimizing in stochastic prediction problems. This characterization is the foundation of the connection between Bregman divergences and iterative centroid-based clustering algorithms noted above.

In this short note, we prove a new characterization of the class of Bregman divergences. This characterization is based on an equality of two common formulations of information content in weighted collections of finite-dimensional vectors.

2. Bregman Divergences Relate Two Informations

Let $[eqn]$ be a probability measure over n points $[eqn]$ . We collect these points into a matrix $[eqn]$ , and in a small abuse of notation, we consider this matrix to be an element of $[eqn]$ . We now define two standard formulations of information, each of which we consider as a function $[eqn]$ . The first formulation compares a weighted sum of strictly convex loss function evaluations on data points to the same loss function evaluated at the data centroid.

Definition 1 (Jensen Gap Information). Let $[eqn]$ be a strictly convex function on $[eqn]$ . The Jensen gap information is the function $[eqn]$ given by

$$[eqn]$$

where $[eqn]$ .

If we define X to be a random vector that takes value $[eqn]$ with probability $[eqn]$ , Jensen’s inequality states that $[eqn]$ , with equality holding only if X is constant (i.e., if there exists i such that $[eqn]$ ). The Jensen gap information is a measure of the difference of the two sides of this inequality; indeed, $[eqn]$ [2,6]. This formulation makes clear that $[eqn]$ is non-negative and that $[eqn]$ if and only if $[eqn]$ whenever $[eqn]$ and $[eqn]$ .

Another standard formulation expresses information content as a weighted mean of divergences of data points from their centroid.

Definition 2 (Divergence). A function $[eqn]$ is a divergence if $[eqn]$ for any $[eqn]$ , with equality if and only if $[eqn]$ .

Definition 3 (Divergence Information). Let d be a divergence. The divergence information is the function $[eqn]$ given by

$$[eqn]$$

where $[eqn]$ .

In this definition, we assume that $[eqn]$ ; as noted by Banerjee et al. [2], this assumption is not restrictive since the set $[eqn]$ can be replaced with the convex hull of the data $[eqn]$ without loss of generality. The divergence information measures the $[eqn]$ -weighted average divergence of $[eqn]$ from the centroid $[eqn]$ . This divergence information is related to the characterization result for Bregman divergences by Banerjee et al. [6]: a divergence d is a Bregman divergence if and only if the vector $[eqn]$ is the unique minimizer of the function $[eqn]$ appearing on the righthand side of Equation (1) for any choice of $[eqn]$ and $[eqn]$ .

There are several important cases in which the Jensen gap information and the divergence information coincide.

Definition 4 (Information Equivalence). We say that a pair $[eqn]$ comprising a strictly convex function $[eqn]$ and a divergence $[eqn]$ satisfies the information equivalence property if, for all $[eqn]$ , it holds that

$$[eqn]$$

A graphical illustration of information equivalence is shown in Figure 1.

Lemma 1 (Information Equivalence with Bregman Divergences [2,6]). The pair $[eqn]$ satisfies the information equivalence property.

The proof is a direct calculation and is provided by Banerjee et al. [2]. When $[eqn]$ and $[eqn]$ is the squared Euclidean distance, the information equivalence property (2) is the identity

$$[eqn]$$

The righthand side of (3) is the weighted sum-of-squares loss of the data points $[eqn]$ with respect to their centroid $[eqn]$ , which is often used in statistical tests and clustering algorithms. Equation (3) asserts that this loss may also be computed from a weighted average of the norms of the data points.

When $[eqn]$ is the probability simplex, $[eqn]$ is the negative entropy, and $[eqn]$ is the KL divergence, the information equivalence property (2) expresses the equality of two equivalent formulations of the mutual information for discrete random variables. Let A and B be discrete random variables on alphabets $[eqn]$ of size n and $[eqn]$ of size ℓ, respectively. Suppose that their joint distribution is $[eqn]$ . Let $[eqn]$ be the vector with entries $[eqn]$ ; then, $[eqn]$ is the marginal distribution of B. The Jensen gap information $[eqn]$ is

$$[eqn]$$

which expresses the mutual information $[eqn]$ between random variables A and B in the entropy-reduction formulation, $[eqn]$ [19]. On the other hand, the divergence information $[eqn]$ is

$$[eqn]$$

which expresses the mutual information $[eqn]$ instead as the weighted sum of KL divergences of $[eqn]$ from $[eqn]$ .

Our contribution in this paper is to prove a converse to Lemma 1: the Bregman divergence $[eqn]$ is the only divergence that satisfies information equivalence with $[eqn]$ .

3. Main Result

Theorem 1.  If the pair $[eqn]$ satisfies the information equivalence property (2), then d is the Bregman divergence induced by ϕ: $[eqn]$ for any $[eqn]$ and $[eqn]$ .

Let $[eqn]$ satisfy information equivalence (2). For any $[eqn]$ and $[eqn]$ , we can write

$$[eqn]$$

for some unknown function $[eqn]$ . We aim to show that $[eqn]$ for all $[eqn]$ and $[eqn]$ .

Our first step is to show that f is an affine function of its first argument $[eqn]$ on $[eqn]$ . To do so, we observe that if $[eqn]$ and $[eqn]$ are such that $[eqn]$ , then we have

$$[eqn]$$

where the first line follows from information equivalence. It follows that

$$[eqn]$$

Fix $[eqn]$ . Let $[eqn]$ , and for any $[eqn]$ let $[eqn]$ . Pick $[eqn]$ sufficiently small that, for all $[eqn]$ , it holds that both $[eqn]$ and $[eqn]$ ; this is possible due to the relative openness of $[eqn]$ . For notational compactness, let $[eqn]$ . Since $[eqn]$ is the intersection of a Euclidean ball with convex set $[eqn]$ , it is also convex.

Consider the function $[eqn]$ given by $[eqn]$ . The condition (5) implies that

$$[eqn]$$

for any $[eqn]$ such that $[eqn]$ .

To show that $[eqn]$ is affine, it suffices to show that the function $[eqn]$ is linear on $[eqn]$ . We do this through two short lemmas. In each, we characterize the behavior of $[eqn]$ on the relative ball $[eqn]$ before extending this characterization to the entire domain $[eqn]$ .

Lemma 2.  For any vector $[eqn]$ and scalar α such that $[eqn]$ , we have $[eqn]$ .

Proof. We will first prove the lemma in the restricted case that $[eqn]$ and $[eqn]$ . By Equation (6), we have that

$$[eqn]$$

from which it follows that $[eqn]$ . Let us now assume that $[eqn]$ but that $[eqn]$ is general; we will then use this to prove the more general setting $[eqn]$ . We proceed by cases.

1. $[eqn]$ . The previous argument implies that $[eqn]$ .
2. $[eqn]$ . Since $[eqn]$ , an application of Equation (6) gives $[eqn]$ ; isolating $[eqn]$ and applying the previous argument proves the case.
3. $[eqn]$ . This case follows by applying the proof of the previous case, replacing $[eqn]$ with $[eqn]$ . Now, assume only that $[eqn]$ . Choose $[eqn]$ so that $[eqn]$ and $[eqn]$ ; $[eqn]$ is one sufficient choice. Then, by our previous argument, we have $[eqn]$ , from which we infer $[eqn]$ . Using this, we can compute $[eqn]$ , which proves the lemma. □

Lemma 3.  The function $[eqn]$ is linear on $[eqn]$ : for any $[eqn]$ and vectors $[eqn]$ such that $[eqn]$ , it holds that

$$[eqn]$$

Proof. Let us first assume that $[eqn]$ and $[eqn]$ . Applying Equation (6) gives

$$[eqn]$$

from which applying Lemma 2 gives the result under these hypotheses.We now consider the general case. For each i, choose $[eqn]$ so that $[eqn]$ and $[eqn]$ . Let $[eqn]$ . Define the vector $[eqn]$ with entries $[eqn]$ . Then, by construction, $[eqn]$ for each i. Applying Lemma 2 and the restricted case above, we can then compute

$$[eqn]$$

which completes the proof. □

Proof of Theorem 1. Fix $[eqn]$ . The preceding lemmas prove that the function $[eqn]$ is linear on $[eqn]$ . Since for constant $[eqn]$ the function f in (4) is a translation of $[eqn]$ in its first argument, it follows that f is affine as a function of its first argument $[eqn]$ . We may therefore write, for all $[eqn]$ and $[eqn]$ ,

$$[eqn]$$

for some functions $[eqn]$ and $[eqn]$ .We now determine these functions. First, since $[eqn]$ is differentiable on $[eqn]$ and $[eqn]$ is affine in $[eqn]$ , $[eqn]$ is differentiable in its first argument on $[eqn]$ . Since d is a divergence, $[eqn]$ is a critical point of the function $[eqn]$ on $[eqn]$ . It follows that $[eqn]$ , the gradient of d with respect to its first argument, is orthogonal to $[eqn]$ at $[eqn]$ :

$$[eqn]$$

for any $[eqn]$ . We can compute $[eqn]$ explicitly; it is $[eqn]$ , which combined with (8) gives

$$[eqn]$$

for any $[eqn]$ and $[eqn]$ .Now, the condition that $[eqn]$ implies that $[eqn]$ . Using Equations (9) and (7), we then compute

$$[eqn]$$

Recalling the definition of f in (4), we conclude that

$$[eqn]$$

which is the Bregman divergence induced by $[eqn]$ . This completes the proof. □

4. Discussion

We have shown that the class of Bregman divergences is the unique class of divergences that induce agreement between the Jensen gap and divergence informations. This result offers some further perspective on the role for Bregman divergences in data clustering and quantization [2]. The Jensen gap information $[eqn]$ is a natural loss function for such tasks, with one motivation as follows. Suppose that we wish to measure the complexity of a set of data points $[eqn]$ with weights $[eqn]$ using a weighted per-observation loss function and a term that depends only on the centroid $[eqn]$ of the data:

$$[eqn]$$

A natural stipulation for the loss function L is that replacing two data points $[eqn]$ and $[eqn]$ with their weighted mean $[eqn]$ should strictly decrease the loss when $[eqn]$ ; this requirement is equivalent to strict convexity of the function $[eqn]$ . If we further require that $[eqn]$ when each row of $[eqn]$ is identical, we find that $[eqn]$ and that our loss function is the Jensen gap information: $[eqn]$ . The present result shows that this natural formulation fully determines the choice of how to perform pairwise comparisons between individual data points; only the corresponding Bregman divergence can serve as a positive-definite comparator that is consistent with the Jensen gap information.

An extension of this result to the setting of Bregman divergences defined on more general spaces, such as Banach spaces [1], would be of considerable interest for problems in functional data clustering [20].